Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Comparing Automatically Detected Reflective Texts with Human Judgements


Published on

Slides from my presentation at the Awareness and Reflection in Technology-Enhanced Learning Workshop at the EC-TEL 2012 Conference. For more information about the workshop and the presentation please visit

Published in: Education, Technology, Business
  • Be the first to comment

  • Be the first to like this

Comparing Automatically Detected Reflective Texts with Human Judgements

  1. 1. Comparing Automatically Detected Reflective Texts with Human Judgements Thomas Daniel Ullmann, Fridolin Wild, Peter Scott KMi - The Open University 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning 18 September 2012
  2. 2. Traditional methods• Questionnaires • Time consuming – Groningen Reflection Ability Scale (GRAS) • Delayed feedback – Reflective Dialogue Rating Scale • Personal nature of• Manual content reflection analysis – Overview see: Dyment, J. E., & O’Connell, T. S. (2011).=> Automated detection of reflection
  3. 3. Related approachesLearning analytics•Associative connection between cuewords and acts of cognition•Machine learning=> related but not on reflection
  4. 4. Theory: Elements of ReflectionDescription of an experience Personal Critical analysisReflection Frame-of-reference Outcome
  5. 5. The ArchitectureUllmann, T.D 2011: An architecture for the automated detection of textual indicators of reflection.
  6. 6. Benefits• Allows the mapping from low level annotations to high level constructs• Knowledge driven• Explanation of inferences
  7. 7. Example ruleFOR ALL sentences of the document:IF sentence contains a nominal subjectAND IF it is a self-referential pronounAND IF the governor of this sentence iscontained in thevocabulary reflective verbsTHEN add fact "Sentence is of type personaluse of reflective vocabulary"
  8. 8. The experiment• Overarching goal: – Evaluating the boundaries of automated detection of reflection• Focus of the paper: – How does automated detection of reflection relate with human judgments of reflection? – What are reasonable weights to parameterise the reflection detector?
  9. 9. Parameterisation=> Weights for the automated detection of reflection
  10. 10. Text corpus• Text corpus: “The Blog Authorship Corpus”• Experiment based on subset: 5176 blog posts• 4.842.295 annotations• 178.504 inferences• Detection: 95 reflective; 54 not reflective ones
  11. 11. Questionnaire
  12. 12. Results
  13. 13. Results
  14. 14. Conclusions• Face values indicate the anticipated difference between reflective texts and not reflective texts• Parameterisation useful
  15. 15. Outlook• Further confirmatory testing• Fine-tuning of weights• Re-evaluation of rules and annotations with larger corpus
  16. 16. Thomas