Despite their importance for educational practice, reflective writings are still manually analysed and assessed, posing a constraint on the use of this educational technique. Recently, research started to investigate automated approaches for analysing reflective writing. Foundational to many automated approaches is the knowledge of words that are important for the genre. This research presents keywords that are specific to several categories of a reflective writing model. These keywords have been derived from eight datasets, which contain several thousand instances using the log-likelihood method. Both performance measures, the accuracy and the Cohen's κ, for these keywords were estimated with ten-fold cross validation. The results reached an accuracy of 0.78 on average for all eight categories and a fair to good inter-rater reliability for most categories even though it did not make use of any sophisticated rule-based mechanisms or machine learning approaches. This research contributes to the development of automated reflective writing analytics that are based on data-driven empirical foundations

1. Reflective writing analytics:
empirically determined
keywords of written reflection
Learning Analytics & Knowledge Conference ‘17
15th of March 2017
Vancouver, Canada
Thomas Daniel Ullmann
Institute of Educational Technology
The Open University, UK

2. LAK17
13-17 March 2017
2
Ullmann, T. D. (2017).
Reflective Writing Analytics
- Empirically Determined
Keywords of Written
Reflection. In Proceedings
of the 7th International
Conference on Learning
Analytics & Knowledge.
Vancouver, BC, Canada:
ACM.

3. Reflective writing
Importance of reflective writing
3
Educational tool to train reflective thinking
Teachers’ pre-service training
Early childhood education
Nursing
Physical therapy
Psychology
Literature
Pharmacy
Business

4. Reflective writing analytics
The manual analysis of reflective
writings
- is time-consuming and costly
- has problems to achieve high inter-
rater reliability
Importance of automated methods to detect reflection
4

5. Reflection detection
Automated methods landscape
5
Dictionary-based
approaches
Rule-based approaches Machine learning based
approaches
Set of rules
Category A
Think, thought
Category B
But, despite
I hadn't
really
thought of it
like this
before but
by
empathising
…
Dictionaries
Inference
engine
IF text
contains 'I'
and 'thought'
THEN add
fact 'personal
thought'
Machine learning
algorithms
Naïve Bayes
Neural Networks
Further algorithms
Further rules
Examples: Bruno, et al. (2011),
Chang et al. (2012), Kann and
Högfeldt (2016), Ullmann (2011)
Examples:Shum, et al. (2016),
Gibson, et al. (2016), Ullmann
(2012)
Ullmann (2015)
Further
categories SVM
See Ullmann, T.D., 2015. Automated detection of reflection in texts - A machine learning based approach.

6. Reflection detection
Automated methods landscape
6
Dictionary-based
approaches
Rule-based approaches Machine learning based
approaches
Set of rules
Category A
Think, thought
Category B
But, despite
I hadn't
really
thought of it
like this
before but
by
empathising
…
Dictionaries
Inference
engine
IF text
contains 'I'
and 'thought'
THEN add
fact 'personal
thought'
Machine learning
algorithms
Naïve Bayes
Neural Networks
Further algorithms
Further rules
Examples: Bruno, et al. (2011),
Chang et al. (2012), Kann and
Högfeldt (2016), Ullmann (2011)
Examples:Shum, et al. (2016),
Gibson, et al. (2016), Ullmann
(2012)
Ullmann (2015)
Further
categories SVM
See Ullmann, T.D., 2015. Automated detection of reflection in texts - A machine learning based approach.

7. Model for reflection detection
Synthesis of 24 models (Ullmann, 2015)
7
Reflection: descriptive/non-reflective and reflective
Description of an experience
Feelings
Personal beliefs
Recognising difficulties/ critical stance
Perspective
Outcome:
- Lessons learned
- Future intentions

8. Keyword approach
● The proposed approach is data-driven (vs. expert-driven)
● The aim is to identify keywords that can be used to automatically
analyse reflective writings
● Method
● 10-fold cross validation
● Identification of keywords on training dataset using the log-likelihood approach
● Determining best candidate keywords on validation dataset
● Evaluating performance of best candidate keyword(s) on test dataset
Outline
8

9. Dataset
Split of dataset
9
Training Test
Validation

10. Calculating ‘keyness’ of words
Class 1 Class 2
Frequency of a word a b
Total c d
10
Log-likelihood statistic
Read and Cressie (1988) cited by Rayson, P. (2008). From key words to key semantic domains. International Journal of
Corpus Linguistics, 13(4). LL is also referred to as the G-test.
















k
i i
i
i
ected
observed
observedLL
1 exp
ln2
In our case:
𝐿𝐿 = 2 × 𝑎 × 𝑙𝑛
𝑎
𝑒1
+ 𝑏 × 𝑙𝑛
𝑏
𝑒2 )/()(
)/()(
2
1
dcbade
dcbace



11. LL example output
Term Class 1 Class 2 Log-likelihood
i 502 313 369
have 112 26 161
me 193 128 133
feel 80 16 122
felt 68 10 115
my 180 155 88
that 278 379 52
better 33 7 49
myself 27 4 46
believe 32 8 44
… … … …
Long list of terms ordered by their log-likelihood
11

12. Determining best candidate example
Itr. Keywords Cohens κ
1 i 0.58
2 i, have 0.49
3 i, have, me 0.52
4 i, have, me, feel 0.51
5 i, have, me, feel, felt 0.51
… … …
Validation dataset
12

13. Determining performance
Text Human Machine Agreement
as i rarely feel confident enough to assert myself it took a lot of courage to stand up for
my learning needs class1 class1 yes
hodston and simpson 2002 contend that understanding our thoughts feelings and
behaviours is essential to provide the best possible care and reach our maximum
potential as nurses class2 class2 yes
critical analysis and reflection are also dependent on self-awareness timmins 2006 class2 class2 yes
work was not going to be rushed off at the last minute it was going to be handed in in
advance class2 class2 yes
and fundamentally i learnt that if you want something done properly do it yourself class1 class1 yes
another similar problem that i encountered was how to tell someone that i didnt like
their idea and that it wasent going to get used class1 class1 yes
if i had ensure that everything along the critical path always ticked along in time and
that all tasks without precursors were done early on then the project would have been
easy come easy go class1 class1 yes
assigning one single person to do this from the start will probably be the best way to do
this class2 class2 yes
learning a foreign language is a complex process affected by an infinite number of
variables and conditions class2 class2 yes
as far as i am concerned the lack of motivation comprehensible input and negotiated
output were the most crucial factors that impeded my progress in learning persian class1 class1 yes
which means i do not know you do not know class2 class1 no
Test dataset
13

14. 10 fold cross validation
14
1st iteration
2nd iteration
3rd iteration
10th iteration
…

15. Dataset
Dataset N Class 1 Class 2
Reflection 2347 603 1744
Experience 3392 1563 1829
Feeling 2672 811 1861
Belief 2303 1188 1115
Difficulty 2717 1392 1325
Perspective 2028 330 1698
Learning 1882 699 1183
Intention 3755 347 3408
Size and class distribution
15

16. Results
Category Nfold Accuracy Cohen's κ
Mean Mean SD Mean SD
Reflection 234.7 0.83 0.03 0.59 0.07
Experience 339.2 0.82 0.02 0.65 0.03
Feeling 267.2 0.79 0.03 0.56 0.05
Belief 230.3 0.70 0.03 0.39 0.06
Difficulty 271.7 0.73 0.03 0.47 0.05
Perspective 202.8 0.74 0.06 0.28 0.04
Learning 188.2 0.67 0.05 0.34 0.06
Intention 375.5 0.93 0.01 0.51 0.10
Average 263.70 0.78 0.03 0.47 0.06
Accuracy and Cohen’s κ
16

17. Best candidate keywords
Category Keywords
Reflection 'I' (100%) and 'me' (60%)
Experience Singular first-person pronouns 'I' (100%) and 'me', (100%);
plural first-person pronoun 'we' (90%); past tense auxiliary verbs
'was' (100%), 'had' (100%), 'were' (90%), and 'did' (50%)
Feeling Singular first-person pronouns 'I' (100%) and 'me' (60%); verbs
'feel' (80%) and 'felt' (60%)
Belief First-person pronouns 'I' (100%), 'my' (80%), and 'it' (50%);
sensing and thinking verbs 'feel' (100%), 'believe' (90%), and
'think' (80%); auxiliary verb 'have' (60%)
Difficulty Conjunctions 'because' (100%), 'but' (100%), 'if' (90%), and
'although' (70%); the nouns 'lack' (90%), 'problems' (80%), and
'situation' (80%); the adjectives 'difficult' (100%), 'due' (100%),
and 'wrong' (80%); the verbs 'trying' (60%), 'felt' (50%), and
'made' (90%); the auxiliary verbs 'did' (100%), 'didn’t' (100%),
'don’t' (60%), 'have' (100%), 'could' (100%), 'would' (100%), and
'may' (100%); the adverbs 'still' (70%), 'not' (100%), and
'however' (100%); and the third-person pronoun 'it' (60%)
17

18. Best candidate keywords
Category Keywords
Perspective Third person pronouns 'they' (100%), 'she' (50%), and 'his'
(50%); the verbs 'felt' (90%), 'said' (80%), and 'understand'
(50%); the auxiliary verbs 'may' (100%), 'might' (50%), and
'would' (50%); the adjective 'aware' (50%); and the conjunction
'that' (80%)
Learning First-person pronouns 'me' (100%) and 'I' (70%); the nouns
'future' (100%), and 'experience' (90%); the verbs 'learnt' (100%)
and 'have' (80%); and 'better' (100%), which was used as either
adjective or adverb
Intention Modal verb 'will' (100%)
18
… continued

19. Conclusion
● Keyword method generated word lists that showed fair to good
reliability for most categories of the reflection detection model.
● Most categories had an accuracy above the baseline.
● Some of the keywords of categories such as ‘Experience’, ‘Reflection’,
and ‘Feeling’ performed better than others, such as ‘Perspective’ and
‘Learning’.
● With a relatively small set of keywords, we can already detect
categories of the reflective writing model.
● Keywords have been determined based on their empirical properties
and not by expert judgement.
● Contribution to a better understanding of reflective writings.
19

20. Thomas Ullmann
thomas.ullmann@open.ac.uk
http://qone.eu/ullmann