Presented at the CALRG seminar of the Open University, UK. Based on: Ullmann, T. D. (2015). Automated detection of reflection in texts. A machine learning based approach. The Open University. Available at http://oro.open.ac.uk/45402/

1. Automated content analysis
of reflective writing
Thomas Ullmann, Institute of Educational Technology
CALRG seminar 05 May 2016 1

2. Overview
●Theory
●Method
●Evaluation
●Conclusion
2

3. Importance of reflection and its detection
Reflection: core to educational practice
● UK Quality Assurance Agency (QAA)
● Organisation for Economic Co-operation and Development (OECD)
● Programme for International Student Assessment (PISA)
Grand challenges in TEL
● E-assessment and automated feedback
3

4. Which side is reflective?
I need to tell her honestly about the
tutorial, the feedback and my
disappointment in myself.
I was immediately embarrassed by my
callous attitude especially when so
many people had died and were
injured.
Finally I believe that throughout these
weeks I have learned some interesting
issues about interactive skills and
cross-cultural communications.
4
I will begin by giving some
background information on the family,
I will then go on to identify the
various stressors and explain how
the framework can be applied.
This week we are performing some
mock appraisal interviews in class,
where I will participate as an
interviewee and an observer.
Hughes states that bed-rails should
be avoided due to the risk of injury
caused when the patient climbs over
them and falls to the floor.
Left or right?

5. Text-based learning analytics
Automated detection of reflective thinking in texts
5
Ullmann, T. D. (2015). Automated detection of reflection in texts. A
machine learning based approach. The Open University. Available
at http://oro.open.ac.uk/45402/
Try the demo: http://qone.eu/reflectr
Reflection Detection
(Classification)
Text as input

6. Theory
6

7. Models to analyse reflective writings
7
Ross (1989), Sparks-Langer and Colto (1991), Gore
and Zeichner (1991), Tsangaridou and O’Sullivan
(1994), Hatton and Smith (1995), Richardson and
Maltby (1995), Pultorak (1996), Hutchinson and Allen
(1997), Scanlan and Chernomas (1997), Taylor (1997),
Valli (1997), Bain et al. (1999), Kim (1999), Duke and
Appleton (2000), Rogers (2001), Bain et al. (2002), Jay
and Johnson (2002), Spalding et al. (2002), MacLellan
(2004), Tillema (2004), Thorpe (2004), Ward and
McCotter (2004), Lee (2005), Korthagen and Vasalos
(2005), Lee (2005), Kansanaho et al. (2005), Kreber
(2005), Wessel and Larin (2006), Mann et al. (2007),
Chretien et al. (2008), Kreber and Castleden (2008),
Minott (2008), Wilson (2008), Gulwadi (2009),
Friedman and Schoen (2009), Le Cornu (2009),
Badger (2010), Granberg (2010), Lambe (2011),
Cohen-Sayag and Fischl (2012), Crawford et al.
(2012), Etscheidt et al. (2012), Leijen et al. (2012),
Corlett (2013), Medwell and Wray (2014), McDonald et
al. (2014), Nguyen et al. (2014), Chaumba (2015), Hill
et al. (2015), and McKay and Dunn (2015)
Sparks- Langer et al. (1990), Wong et al.
(1995), Sumsion and Fleet (1996), McCollum
(1997), Kember et al. (1999), Hawkes and
Romiszowski (2001), Hawkes (2001, 2006),
Fund et al. (2002), Hamann (2002), Pee et al.
(2002), Williams (2000), Boenink et al. (2004),
O'Connell and Dyment (2004), Plack et al.
(2005), Ballard (2006), Mansvelder-Lonaryoux
(2006), Mansvelder-Longayroux et al. (2007),
Abou Baker El-Dib (2007), Chirema (2007),
Plack et al. (2007), Kember et al. (2008),
Wallman et al. (2008), Chamoso and Caceres
(2009), Findlay et al. (2010), Lai and Calandra
(2010), Bell et al. (2011), Clarkeburn and
Kettula (2011), Findlay et al. (2011), Fischer et
al. (2011), Birney (2012), Ip et al. (2012), Wald et
al. (2012), Mena-Marcos et al. (2013), Poom-
Valickis and Mathews (2013), Poldner et al.
(2014), Prilla and Renner (2014)

8. Models to analyse reflective writings
8

9. Qualities of reflective writings
● Depth dimension (hierarchy of levels)
● Breadth dimension (describes types of reflection)
9
descriptive reflective
?

10. Synthesis of common categories
Author(s) Experience Feelings Personal Critical Perspective Outcome
Sparks-Langer et al. (1990) ✔ ✔ ✔ ✔
Wong et al. (1995) ✔ ✔ ✔ ✔ ✔
McCollum (1997) ✔ ✓ ✓ ✔ ✔
Kember et al. (1999) ✔ ✔ ✔ ✔ ✔
Fund et al. (2002) ✔ ✔ ✔ ✔ ✔ ✓
Hamann (2002) ✔ ✔ ✔
Pee et al. (2002) ✔ ✔ ✔ ✔
Williams et al. (2002) ✔ ✔ ✔ ✔ ✔
Boenink et al. (2004) ✔ ✔ ✔ ✔
O’Connell and Dyment (2004) ✔ ✔ ✔
Plack et al. (2005) ✔ ✔ ✔ ✔ ✔ ✔
Ballard (2006) ✔ ✔ ✓ ✔
Mansvelder-Longayroux (2006,2007) ✔ ✓ ✔ ✔ ✔
Plack et al. (2007) ✔ ✔ ✔ ✔ ✔ ✔
Kember et al. (2008) ✔ ✔ ✔ ✔ ✔
Wallman et al. (2008) ✔ ✔ ✔ ✔ ✔ ✔
Chamoso and Cáceres (2009) ✔ ✓ ✔ ✔
Lai and Calandra (2010) ✔ ✔ ✔ ✔ ✔ ✔
Fischer et al. (2011) ✔ ✔ ✔ ✔ ✔
Birney (2012) ✔ ✔ ✔ ✔ ✔ ✔
Wald et al. (2012) ✔ ✔ ✔ ✔ ✔ ✔
Mena-Marcos et al. (2013) ✓ ✔ ✔
Poldner et al. (2014) ✔ ✓ ✔ ✔
Prilla and Renner (2014) ✔ ✔ ✓ ✔ ✔ ✔
10

11. Model for reflection detection
●Depth of reflection
● Descriptive vs. reflective
●Breadth of reflection
● Description of an experience: Subject matter of the reflective writing
● Feelings: Doubts, uncertainty, frustration, surprise, excitement, etc.
● Personal: One's assumptions, beliefs, knowledge of self
● Critical stance: Critical mindset; awareness of problems
● Perspective: Awareness of other perspectives
● Outcome: Retrospective: lessons learned; prospective: future intentions
11

12. Claims
1. Machine learning algorithms can be used to
distinguish between descriptive and reflective
text segments (RQ1)
2. Machine learning algorithms can be used to
detect common categories of reflective writings
(RQ2)
12

13. Method

14. Method overview
Dataset
Training data Test data
Models Assessment
Text collection
Annotation Task
Annotated units
EvaluationData generation
14

15. Dataset generation process
15
Text collection
Identifcation of
suitable text collections
Sampling of
text collection
Unitising text collection
Dataset of unlabelled units
Annotation task
Task design Pilots
Quality standard
Rated units
Dataset
Reliability
Validity
Annotated units

16. Datasets
16

17. Dataset reliability estimates
17

18. Reliability annotation task
Simple majority
18

19. Model validation
19
Correlation between reflection indicator and common categories

20. Research design
Dataset for machine learning
Training data Test data
Model selection Model assessment
Dataset of labelled units
Data pre-processing Splitting
Feature construction
Feature selection
Oversampled dataset
Resampling
Model tuning
Original class distribution
Pre-processsing Machine learning
20

21. Evaluation
21

22. Instantiation of method for RQ1
Can machine learning be used to distinguish between
descriptive and reflective text segments?
22
Rule-based models
Tree-based models
High performance
Reflection
Datasets Research design
Research
question
RQ1 I1
RQ1 I2
RQ1 I3
Three lines of investigation to answer research question 1

23. RQ1 Results
Comparison of the three lines of investigation
23

24. Instantiation of method for RQ2
Can machine learning algorithms be used to detect common
categories of reflective writing?
24
Experience
Feelings
Personal
Critical stance
Perspective
Outcome
Datasets Research design
Research
question
High performance
models
RQ2 Exp.
RQ2 Feel.
RQ2 Pers.
RQ2 Crit.
RQ2 Persp.
RQ2 Out.

25. RQ2 Results
Indicator N Cohen’s k % Landis & Koch BM % CA BM
Experience 654 0.83 0.92 Almost perfect Top
Feelings 521 0.73 0.88 Substantial Middle
Beliefs 449 0.66 0.83 Substantial Middle
Difficulties 526 0.60 0.80 Moderate Middle
Perspective 396 0.55 0.88 Moderate Middle
Intention 727 0.71 0.95 Substantial Top
Learning 364 0.63 0.83 Substantial Middle
Reflection 456 0.70 0.89 Substantial Middle
Automated detection of common categories of reflection
25

26. Comparison of model and dataset
Per cent agreement
26

27. Conclusion

28. Conclusion
Machine learning algorithms can be used to distinguish between
descriptive and reflective text segments
Machine learning algorithms can be used to detect common
categories of reflective writings
28

29. Limitations
● Investigated language
● Investigated unit of analysis
29

30. H818 The networked practitioner
Introduction to reflective writing to support TMAs and EMA
30

31. Text-based learning analytics
Automated detection of reflective thinking in texts
31
Ullmann, T. D. (2015). Automated detection of reflection in texts. A
machine learning based approach. The Open University. Available
at http://oro.open.ac.uk/45402/
Try the demo: http://qone.eu/reflectr
Reflection Detection
(Classification)
Text as input

32. Thank you
32

33. See for a different approach
Ullmann, T. D. (2015). Keywords of
written reflection - a comparison between
reflective and descriptive datasets. In
Proceedings of the 5th Workshop on
Awareness and Reflection in Technology
Enhanced Learning (Vol. 1465, pp. 83–
96). Toledo, Spain
Keywords of written reflection
33