Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automated content analysis of reflective writing

269 views

Published on

Presented at the CALRG seminar of the Open University, UK. Based on:
Ullmann, T. D. (2015). Automated detection of reflection in texts. A machine learning based approach. The Open University. Available at http://oro.open.ac.uk/45402/

Published in: Science
  • Be the first to comment

Automated content analysis of reflective writing

  1. 1. Automated content analysis of reflective writing Thomas Ullmann, Institute of Educational Technology CALRG seminar 05 May 2016 1
  2. 2. Overview ●Theory ●Method ●Evaluation ●Conclusion 2
  3. 3. Importance of reflection and its detection Reflection: core to educational practice ● UK Quality Assurance Agency (QAA) ● Organisation for Economic Co-operation and Development (OECD) ● Programme for International Student Assessment (PISA) Grand challenges in TEL ● E-assessment and automated feedback 3
  4. 4. Which side is reflective? I need to tell her honestly about the tutorial, the feedback and my disappointment in myself. I was immediately embarrassed by my callous attitude especially when so many people had died and were injured. Finally I believe that throughout these weeks I have learned some interesting issues about interactive skills and cross-cultural communications. 4 I will begin by giving some background information on the family, I will then go on to identify the various stressors and explain how the framework can be applied. This week we are performing some mock appraisal interviews in class, where I will participate as an interviewee and an observer. Hughes states that bed-rails should be avoided due to the risk of injury caused when the patient climbs over them and falls to the floor. Left or right?
  5. 5. Text-based learning analytics Automated detection of reflective thinking in texts 5 Ullmann, T. D. (2015). Automated detection of reflection in texts. A machine learning based approach. The Open University. Available at http://oro.open.ac.uk/45402/ Try the demo: http://qone.eu/reflectr Reflection Detection (Classification) Text as input
  6. 6. Theory 6
  7. 7. Models to analyse reflective writings 7 Ross (1989), Sparks-Langer and Colto (1991), Gore and Zeichner (1991), Tsangaridou and O’Sullivan (1994), Hatton and Smith (1995), Richardson and Maltby (1995), Pultorak (1996), Hutchinson and Allen (1997), Scanlan and Chernomas (1997), Taylor (1997), Valli (1997), Bain et al. (1999), Kim (1999), Duke and Appleton (2000), Rogers (2001), Bain et al. (2002), Jay and Johnson (2002), Spalding et al. (2002), MacLellan (2004), Tillema (2004), Thorpe (2004), Ward and McCotter (2004), Lee (2005), Korthagen and Vasalos (2005), Lee (2005), Kansanaho et al. (2005), Kreber (2005), Wessel and Larin (2006), Mann et al. (2007), Chretien et al. (2008), Kreber and Castleden (2008), Minott (2008), Wilson (2008), Gulwadi (2009), Friedman and Schoen (2009), Le Cornu (2009), Badger (2010), Granberg (2010), Lambe (2011), Cohen-Sayag and Fischl (2012), Crawford et al. (2012), Etscheidt et al. (2012), Leijen et al. (2012), Corlett (2013), Medwell and Wray (2014), McDonald et al. (2014), Nguyen et al. (2014), Chaumba (2015), Hill et al. (2015), and McKay and Dunn (2015) Sparks- Langer et al. (1990), Wong et al. (1995), Sumsion and Fleet (1996), McCollum (1997), Kember et al. (1999), Hawkes and Romiszowski (2001), Hawkes (2001, 2006), Fund et al. (2002), Hamann (2002), Pee et al. (2002), Williams (2000), Boenink et al. (2004), O'Connell and Dyment (2004), Plack et al. (2005), Ballard (2006), Mansvelder-Lonaryoux (2006), Mansvelder-Longayroux et al. (2007), Abou Baker El-Dib (2007), Chirema (2007), Plack et al. (2007), Kember et al. (2008), Wallman et al. (2008), Chamoso and Caceres (2009), Findlay et al. (2010), Lai and Calandra (2010), Bell et al. (2011), Clarkeburn and Kettula (2011), Findlay et al. (2011), Fischer et al. (2011), Birney (2012), Ip et al. (2012), Wald et al. (2012), Mena-Marcos et al. (2013), Poom- Valickis and Mathews (2013), Poldner et al. (2014), Prilla and Renner (2014)
  8. 8. Models to analyse reflective writings 8
  9. 9. Qualities of reflective writings ● Depth dimension (hierarchy of levels) ● Breadth dimension (describes types of reflection) 9 descriptive reflective ?
  10. 10. Synthesis of common categories Author(s) Experience Feelings Personal Critical Perspective Outcome Sparks-Langer et al. (1990) ✔ ✔ ✔ ✔ Wong et al. (1995) ✔ ✔ ✔ ✔ ✔ McCollum (1997) ✔ ✓ ✓ ✔ ✔ Kember et al. (1999) ✔ ✔ ✔ ✔ ✔ Fund et al. (2002) ✔ ✔ ✔ ✔ ✔ ✓ Hamann (2002) ✔ ✔ ✔ Pee et al. (2002) ✔ ✔ ✔ ✔ Williams et al. (2002) ✔ ✔ ✔ ✔ ✔ Boenink et al. (2004) ✔ ✔ ✔ ✔ O’Connell and Dyment (2004) ✔ ✔ ✔ Plack et al. (2005) ✔ ✔ ✔ ✔ ✔ ✔ Ballard (2006) ✔ ✔ ✓ ✔ Mansvelder-Longayroux (2006,2007) ✔ ✓ ✔ ✔ ✔ Plack et al. (2007) ✔ ✔ ✔ ✔ ✔ ✔ Kember et al. (2008) ✔ ✔ ✔ ✔ ✔ Wallman et al. (2008) ✔ ✔ ✔ ✔ ✔ ✔ Chamoso and Cáceres (2009) ✔ ✓ ✔ ✔ Lai and Calandra (2010) ✔ ✔ ✔ ✔ ✔ ✔ Fischer et al. (2011) ✔ ✔ ✔ ✔ ✔ Birney (2012) ✔ ✔ ✔ ✔ ✔ ✔ Wald et al. (2012) ✔ ✔ ✔ ✔ ✔ ✔ Mena-Marcos et al. (2013) ✓ ✔ ✔ Poldner et al. (2014) ✔ ✓ ✔ ✔ Prilla and Renner (2014) ✔ ✔ ✓ ✔ ✔ ✔ 10
  11. 11. Model for reflection detection ●Depth of reflection ● Descriptive vs. reflective ●Breadth of reflection ● Description of an experience: Subject matter of the reflective writing ● Feelings: Doubts, uncertainty, frustration, surprise, excitement, etc. ● Personal: One's assumptions, beliefs, knowledge of self ● Critical stance: Critical mindset; awareness of problems ● Perspective: Awareness of other perspectives ● Outcome: Retrospective: lessons learned; prospective: future intentions 11
  12. 12. Claims 1. Machine learning algorithms can be used to distinguish between descriptive and reflective text segments (RQ1) 2. Machine learning algorithms can be used to detect common categories of reflective writings (RQ2) 12
  13. 13. Method
  14. 14. Method overview Dataset Training data Test data Models Assessment Text collection Annotation Task Annotated units EvaluationData generation 14
  15. 15. Dataset generation process 15 Text collection Identifcation of suitable text collections Sampling of text collection Unitising text collection Dataset of unlabelled units Annotation task Task design Pilots Quality standard Rated units Dataset Reliability Validity Annotated units
  16. 16. Datasets 16
  17. 17. Dataset reliability estimates 17
  18. 18. Reliability annotation task Simple majority 18
  19. 19. Model validation 19 Correlation between reflection indicator and common categories
  20. 20. Research design Dataset for machine learning Training data Test data Model selection Model assessment Dataset of labelled units Data pre-processing Splitting Feature construction Feature selection Oversampled dataset Resampling Model tuning Original class distribution Pre-processsing Machine learning 20
  21. 21. Evaluation 21
  22. 22. Instantiation of method for RQ1 Can machine learning be used to distinguish between descriptive and reflective text segments? 22 Rule-based models Tree-based models High performance Reflection Datasets Research design Research question RQ1 I1 RQ1 I2 RQ1 I3 Three lines of investigation to answer research question 1
  23. 23. RQ1 Results Comparison of the three lines of investigation 23
  24. 24. Instantiation of method for RQ2 Can machine learning algorithms be used to detect common categories of reflective writing? 24 Experience Feelings Personal Critical stance Perspective Outcome Datasets Research design Research question High performance models RQ2 Exp. RQ2 Feel. RQ2 Pers. RQ2 Crit. RQ2 Persp. RQ2 Out.
  25. 25. RQ2 Results Indicator N Cohen’s k % Landis & Koch BM % CA BM Experience 654 0.83 0.92 Almost perfect Top Feelings 521 0.73 0.88 Substantial Middle Beliefs 449 0.66 0.83 Substantial Middle Difficulties 526 0.60 0.80 Moderate Middle Perspective 396 0.55 0.88 Moderate Middle Intention 727 0.71 0.95 Substantial Top Learning 364 0.63 0.83 Substantial Middle Reflection 456 0.70 0.89 Substantial Middle Automated detection of common categories of reflection 25
  26. 26. Comparison of model and dataset Per cent agreement 26
  27. 27. Conclusion
  28. 28. Conclusion Machine learning algorithms can be used to distinguish between descriptive and reflective text segments Machine learning algorithms can be used to detect common categories of reflective writings 28
  29. 29. Limitations ● Investigated language ● Investigated unit of analysis 29
  30. 30. H818 The networked practitioner Introduction to reflective writing to support TMAs and EMA 30
  31. 31. Text-based learning analytics Automated detection of reflective thinking in texts 31 Ullmann, T. D. (2015). Automated detection of reflection in texts. A machine learning based approach. The Open University. Available at http://oro.open.ac.uk/45402/ Try the demo: http://qone.eu/reflectr Reflection Detection (Classification) Text as input
  32. 32. Thank you 32
  33. 33. See for a different approach Ullmann, T. D. (2015). Keywords of written reflection - a comparison between reflective and descriptive datasets. In Proceedings of the 5th Workshop on Awareness and Reflection in Technology Enhanced Learning (Vol. 1465, pp. 83– 96). Toledo, Spain Keywords of written reflection 33

×