Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Reflective writing analytics: empirically determined keywords of written reflection

1,027 views

Published on

Despite their importance for educational practice, reflective writings are still manually analysed and assessed, posing a constraint on the use of this educational technique. Recently, research started to investigate automated approaches for analysing reflective writing. Foundational to many automated approaches is the knowledge of words that are important for the genre. This research presents keywords that are specific to several categories of a reflective writing model. These keywords have been derived from eight datasets, which contain several thousand instances using the log-likelihood method. Both performance measures, the accuracy and the Cohen's κ, for these keywords were estimated with ten-fold cross validation. The results reached an accuracy of 0.78 on average for all eight categories and a fair to good inter-rater reliability for most categories even though it did not make use of any sophisticated rule-based mechanisms or machine learning approaches. This research contributes to the development of automated reflective writing analytics that are based on data-driven empirical foundations

Published in: Education
  • I am making a good MONEY (500$ to 700$ / hr )online on my Ipad .Last month my pay check of nearly 30 k$.This online work is like draw straight-arrow and earn money.Do not go to office.I do not claim to be others,I just work.You will call yourself after doing this JOB,It's a REAL job.Will be very lucky to refer to this WEBSITE.I hope,you can find something,Simply go to the below SITE. ♣♣♣ http://t.cn/AisJWUCf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Can you earn $7000 a month from home? Are you feeling trapped by your life? Stuck in a dead-end job you hate, but too scared to call it quits, because after all, the rent's due on the first of the month, right? Are you ready to change your life for the better? ➤➤ http://ishbv.com/ezpayjobs/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • As a single mother every little bit counts! This has been such a great way for me to earn extra money. As a single mother every little bit counts! Finally, a vehicle for making some honest to goodness real money to make life easier and happier now that I don't have to pull my hair out budgeting every penny every day.Thanks for the rainbow in my sky. ●●● https://tinyurl.com/vd3y33w
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Reflective writing analytics: empirically determined keywords of written reflection

  1. 1. Reflective writing analytics: empirically determined keywords of written reflection Learning Analytics & Knowledge Conference ‘17 15th of March 2017 Vancouver, Canada Thomas Daniel Ullmann Institute of Educational Technology The Open University, UK
  2. 2. LAK17 13-17 March 2017 2 Ullmann, T. D. (2017). Reflective Writing Analytics - Empirically Determined Keywords of Written Reflection. In Proceedings of the 7th International Conference on Learning Analytics & Knowledge. Vancouver, BC, Canada: ACM.
  3. 3. Reflective writing Importance of reflective writing 3 Educational tool to train reflective thinking Teachers’ pre-service training Early childhood education Nursing Physical therapy Psychology Literature Pharmacy Business
  4. 4. Reflective writing analytics The manual analysis of reflective writings - is time-consuming and costly - has problems to achieve high inter- rater reliability Importance of automated methods to detect reflection 4
  5. 5. Reflection detection Automated methods landscape 5 Dictionary-based approaches Rule-based approaches Machine learning based approaches Set of rules Category A Think, thought Category B But, despite I hadn't really thought of it like this before but by empathising … Dictionaries Inference engine IF text contains 'I' and 'thought' THEN add fact 'personal thought' Machine learning algorithms Naïve Bayes Neural Networks Further algorithms Further rules Examples: Bruno, et al. (2011), Chang et al. (2012), Kann and Högfeldt (2016), Ullmann (2011) Examples:Shum, et al. (2016), Gibson, et al. (2016), Ullmann (2012) Ullmann (2015) Further categories SVM See Ullmann, T.D., 2015. Automated detection of reflection in texts - A machine learning based approach.
  6. 6. Reflection detection Automated methods landscape 6 Dictionary-based approaches Rule-based approaches Machine learning based approaches Set of rules Category A Think, thought Category B But, despite I hadn't really thought of it like this before but by empathising … Dictionaries Inference engine IF text contains 'I' and 'thought' THEN add fact 'personal thought' Machine learning algorithms Naïve Bayes Neural Networks Further algorithms Further rules Examples: Bruno, et al. (2011), Chang et al. (2012), Kann and Högfeldt (2016), Ullmann (2011) Examples:Shum, et al. (2016), Gibson, et al. (2016), Ullmann (2012) Ullmann (2015) Further categories SVM See Ullmann, T.D., 2015. Automated detection of reflection in texts - A machine learning based approach.
  7. 7. Model for reflection detection Synthesis of 24 models (Ullmann, 2015) 7 Reflection: descriptive/non-reflective and reflective Description of an experience Feelings Personal beliefs Recognising difficulties/ critical stance Perspective Outcome: - Lessons learned - Future intentions
  8. 8. Keyword approach ● The proposed approach is data-driven (vs. expert-driven) ● The aim is to identify keywords that can be used to automatically analyse reflective writings ● Method ● 10-fold cross validation ● Identification of keywords on training dataset using the log-likelihood approach ● Determining best candidate keywords on validation dataset ● Evaluating performance of best candidate keyword(s) on test dataset Outline 8
  9. 9. Dataset Split of dataset 9 Training Test Validation
  10. 10. Calculating ‘keyness’ of words Class 1 Class 2 Frequency of a word a b Total c d 10 Log-likelihood statistic Read and Cressie (1988) cited by Rayson, P. (2008). From key words to key semantic domains. International Journal of Corpus Linguistics, 13(4). LL is also referred to as the G-test.                 k i i i i ected observed observedLL 1 exp ln2 In our case: 𝐿𝐿 = 2 × 𝑎 × 𝑙𝑛 𝑎 𝑒1 + 𝑏 × 𝑙𝑛 𝑏 𝑒2 )/()( )/()( 2 1 dcbade dcbace  
  11. 11. LL example output Term Class 1 Class 2 Log-likelihood i 502 313 369 have 112 26 161 me 193 128 133 feel 80 16 122 felt 68 10 115 my 180 155 88 that 278 379 52 better 33 7 49 myself 27 4 46 believe 32 8 44 … … … … Long list of terms ordered by their log-likelihood 11
  12. 12. Determining best candidate example Itr. Keywords Cohens κ 1 i 0.58 2 i, have 0.49 3 i, have, me 0.52 4 i, have, me, feel 0.51 5 i, have, me, feel, felt 0.51 … … … Validation dataset 12
  13. 13. Determining performance Text Human Machine Agreement as i rarely feel confident enough to assert myself it took a lot of courage to stand up for my learning needs class1 class1 yes hodston and simpson 2002 contend that understanding our thoughts feelings and behaviours is essential to provide the best possible care and reach our maximum potential as nurses class2 class2 yes critical analysis and reflection are also dependent on self-awareness timmins 2006 class2 class2 yes work was not going to be rushed off at the last minute it was going to be handed in in advance class2 class2 yes and fundamentally i learnt that if you want something done properly do it yourself class1 class1 yes another similar problem that i encountered was how to tell someone that i didnt like their idea and that it wasent going to get used class1 class1 yes if i had ensure that everything along the critical path always ticked along in time and that all tasks without precursors were done early on then the project would have been easy come easy go class1 class1 yes assigning one single person to do this from the start will probably be the best way to do this class2 class2 yes learning a foreign language is a complex process affected by an infinite number of variables and conditions class2 class2 yes as far as i am concerned the lack of motivation comprehensible input and negotiated output were the most crucial factors that impeded my progress in learning persian class1 class1 yes which means i do not know you do not know class2 class1 no Test dataset 13
  14. 14. 10 fold cross validation 14 1st iteration 2nd iteration 3rd iteration 10th iteration …
  15. 15. Dataset Dataset N Class 1 Class 2 Reflection 2347 603 1744 Experience 3392 1563 1829 Feeling 2672 811 1861 Belief 2303 1188 1115 Difficulty 2717 1392 1325 Perspective 2028 330 1698 Learning 1882 699 1183 Intention 3755 347 3408 Size and class distribution 15
  16. 16. Results Category Nfold Accuracy Cohen's κ Mean Mean SD Mean SD Reflection 234.7 0.83 0.03 0.59 0.07 Experience 339.2 0.82 0.02 0.65 0.03 Feeling 267.2 0.79 0.03 0.56 0.05 Belief 230.3 0.70 0.03 0.39 0.06 Difficulty 271.7 0.73 0.03 0.47 0.05 Perspective 202.8 0.74 0.06 0.28 0.04 Learning 188.2 0.67 0.05 0.34 0.06 Intention 375.5 0.93 0.01 0.51 0.10 Average 263.70 0.78 0.03 0.47 0.06 Accuracy and Cohen’s κ 16
  17. 17. Best candidate keywords Category Keywords Reflection 'I' (100%) and 'me' (60%) Experience Singular first-person pronouns 'I' (100%) and 'me', (100%); plural first-person pronoun 'we' (90%); past tense auxiliary verbs 'was' (100%), 'had' (100%), 'were' (90%), and 'did' (50%) Feeling Singular first-person pronouns 'I' (100%) and 'me' (60%); verbs 'feel' (80%) and 'felt' (60%) Belief First-person pronouns 'I' (100%), 'my' (80%), and 'it' (50%); sensing and thinking verbs 'feel' (100%), 'believe' (90%), and 'think' (80%); auxiliary verb 'have' (60%) Difficulty Conjunctions 'because' (100%), 'but' (100%), 'if' (90%), and 'although' (70%); the nouns 'lack' (90%), 'problems' (80%), and 'situation' (80%); the adjectives 'difficult' (100%), 'due' (100%), and 'wrong' (80%); the verbs 'trying' (60%), 'felt' (50%), and 'made' (90%); the auxiliary verbs 'did' (100%), 'didn’t' (100%), 'don’t' (60%), 'have' (100%), 'could' (100%), 'would' (100%), and 'may' (100%); the adverbs 'still' (70%), 'not' (100%), and 'however' (100%); and the third-person pronoun 'it' (60%) 17
  18. 18. Best candidate keywords Category Keywords Perspective Third person pronouns 'they' (100%), 'she' (50%), and 'his' (50%); the verbs 'felt' (90%), 'said' (80%), and 'understand' (50%); the auxiliary verbs 'may' (100%), 'might' (50%), and 'would' (50%); the adjective 'aware' (50%); and the conjunction 'that' (80%) Learning First-person pronouns 'me' (100%) and 'I' (70%); the nouns 'future' (100%), and 'experience' (90%); the verbs 'learnt' (100%) and 'have' (80%); and 'better' (100%), which was used as either adjective or adverb Intention Modal verb 'will' (100%) 18 … continued
  19. 19. Conclusion ● Keyword method generated word lists that showed fair to good reliability for most categories of the reflection detection model. ● Most categories had an accuracy above the baseline. ● Some of the keywords of categories such as ‘Experience’, ‘Reflection’, and ‘Feeling’ performed better than others, such as ‘Perspective’ and ‘Learning’. ● With a relatively small set of keywords, we can already detect categories of the reflective writing model. ● Keywords have been determined based on their empirical properties and not by expert judgement. ● Contribution to a better understanding of reflective writings. 19
  20. 20. Thomas Ullmann thomas.ullmann@open.ac.uk http://qone.eu/ullmann

×