Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automated Evaluation of Comments in a MOOC Discussion Forum

152 views

Published on

My talk at FutureLearn Academic Network day at CALRG Conference 2017, Open University, Milton Keynes, 16 June 2017.

http://cloudworks.ac.uk/cloudscape/view/3004

Published in: Education
  • Be the first to comment

  • Be the first to like this

Automated Evaluation of Comments in a MOOC Discussion Forum

  1. 1. Automated Evaluation of Comments in a MOOC Discussion Forum Tim O’Riordan Web and Internet Science (WAIS) www.ecs.soton.ac.uk/people/tor1w07 @studywbv
  2. 2. Motivation Image: Grumpy MOOC cat/ Ryan Tracey, cc-by-nc-nd, © 2014 Automated Evaluation of Comments in a MOOC Discussion Forum
  3. 3. Motivation Image: NodeXL Twitter Search #dataviz/ Marc Smith, cc-by, © 2013 Automated Evaluation of Comments in a MOOC Discussion Forum
  4. 4. Content analysis Image: ALTC2015, Association for Learning Technology ©2015, CC BY-NC-SA 2.0 Automated Evaluation of Comments in a MOOC Discussion Forum
  5. 5. Critical thinking ”...reasonable and reflective thinking that is focused upon deciding what to do or believe." S. P. Norris & R. H. Ennis, Evaluating Critical Thinking. The Practitioners’ Guide to Teaching Thinking Series. Pacific Grove, CA: Midwest Publications, 1989. Automated Evaluation of Comments in a MOOC Discussion Forum
  6. 6. Method Categories Bloom’s Taxonomy 0 Off-topic 1 Remember 2 Understand 3 Apply 4 Analyse 5 Evaluate 6 Create Cognitive Presence (CoI) 0 Off-topic 1 Triggering 2 Exploration 3 Integration 4 Resolution Content Analysis Methods Automated Evaluation of Comments in a MOOC Discussion Forum D. R. Krathwohl, “A Revision of Bloom’s Taxonomy: An Overview,” Theory Pract., vol. 41, no. 4, pp. 212–218, 2002. D. R. Garrison, T. Anderson, and W. Archer, “Critical thinking, cognitive presence, and computer conferencing in distance education,” Am. J. Distance Educ., vol. 15, no. 1, pp. 7–23, 2001.
  7. 7. • 500 comments each MOOC • Rated according to 2 methods by 7 raters • Linguistic Inquiry and Word Count (LIWC 2015) Image: University of Southampton, Understanding Language, Exploring Oceans and Contract Management MOOCs. FutureLearn Ltd 2015. Automated Evaluation of Comments in a MOOC Discussion Forum
  8. 8. Results Correlation between Bloom and Cognitive Presence r = 0.909 p = <0.001 Automated Evaluation of Comments in a MOOC Discussion Forum
  9. 9. Results Correlation between CP and Word Count r = 0.704 p = <0.001 Automated Evaluation of Comments in a MOOC Discussion Forum 0 1 2 3 4 Cognitive Presence 0 50 100 150 200 WordCount Sheet 2
  10. 10. 0 1 2 3 4 Cognitive Presence 0 10 20 30 40 50 1stpersonpronoun Sheet 3 Results Correlation between CP and 1st person singular r = -0.317 p = <0.001 Automated Evaluation of Comments in a MOOC Discussion Forum
  11. 11. Images: Weka Logo/Weka interface, University of Waikato ©2015 Automated Evaluation of Comments in a MOOC Discussion Forum
  12. 12. Machine Learning – CP • Classifiers: Naive Bayes, J48, ZeroR, Random Forest • 13 attributes: word count, WPS, Sixltr, ppron, auxverb, conj, posemo, negemo, cogproc, cause, differ, affiliation, power • <50 word comments • Accuracy: 69% • Cohen’s k: 0.612 Automated Evaluation of Comments in a MOOC Discussion Forum
  13. 13. Predicted Actual Off-topic Triggering Exploration Integration Resolution Off-topic 60 0 0 0 0 Triggering 0 58 0 0 12 Exploration 3 9 28 11 8 Integration 1 3 15 23 19 Resolution 1 2 5 14 38 Confusion matrix for best model – Cognitive presence Machine Learning – CP Automated Evaluation of Comments in a MOOC Discussion Forum
  14. 14. User study Image: University of Southampton and MOOCAP, Digital Accessibility MOOC. FutureLearn Ltd 2017 Automated Evaluation of Comments in a MOOC Discussion Forum
  15. 15. User study Image: Automatic rating of all comments in all steps for Week 1, DA MOOC, 2017. Automated Evaluation of Comments in a MOOC Discussion Forum
  16. 16. User study Image: Automatic rating of all comments in all steps for Week 1.18, DA MOOC, 2017. Automated Evaluation of Comments in a MOOC Discussion Forum
  17. 17. User study Image: Automatic rating of all comments in all steps for Week 1.17, DA MOOC, 2017. Automated Evaluation of Comments in a MOOC Discussion Forum
  18. 18. Initial outcomes • Questioning importance of monitoring critical thinking • But...seek reflective, high level comments • Automatic ratings considered reasonably accurate • Value feedback that facilitates filtering high volumes Automated Evaluation of Comments in a MOOC Discussion Forum
  19. 19. Is critical thinking important? “I found that it kind of focused on critical thinking, which I hadn’t really thought about. And what would be interesting would be to, when you’re designing the course to be thinking in those terms. I suppose when you’re designing a course you’re looking at the getting material across and doing the job or whatever, but you’re not thinking critical thinking as a way of engagement. You’re more worried about what the research says about MOOCs.” Automated Evaluation of Comments in a MOOC Discussion Forum
  20. 20. Is critical thinking important? “…in a course like this where it is largely, I suppose partly informative...and partly designed about quick takeaway, I think it might be a secondary concern…” Automated Evaluation of Comments in a MOOC Discussion Forum
  21. 21. Is critical thinking important? “...depends on to what level of critical thinking you’re trying to establish, but you need to have … some sort of element to show where the learner’s got to.” Automated Evaluation of Comments in a MOOC Discussion Forum
  22. 22. Seek reflective comments “I am actually looking for comments which are reflective in a sense and maybe come up with extra information but also maybe ask questions… these are the best to show that they have learned something.” Automated Evaluation of Comments in a MOOC Discussion Forum
  23. 23. Value of automatic rating “I think actually it was about right; it was enough to kind of get a feel for what was going on and particularly for the visualisation, to actually see it was excellent, it was really helpful.” Automated Evaluation of Comments in a MOOC Discussion Forum
  24. 24. Filtering comments “MOOC’s are always a challenge to be able to look at...even if it’s not critical thinking you would like to look at the comments in a more efficient way... It will save time and it will give a highlight to things that are important.” Automated Evaluation of Comments in a MOOC Discussion Forum
  25. 25. Next steps • Reduce number of categories • Explore method to include <50 word comments • Develop a real time application Automated Evaluation of Comments in a MOOC Discussion Forum
  26. 26. @studywbv tim.oriordan@soton.ac.uk Thank you… timswww.wordpress.com Automated Evaluation of Comments in a MOOC Discussion Forum

×