Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Automated Evaluation of
Comments in a
MOOC Discussion Forum
Tim O’Riordan
Web and Internet Science (WAIS)
www.ecs.soton.ac...
Motivation
Image: Grumpy MOOC cat/ Ryan Tracey, cc-by-nc-nd, © 2014 Automated Evaluation of Comments
in a MOOC Discussion ...
Motivation
Image: NodeXL Twitter Search #dataviz/ Marc Smith, cc-by, © 2013 Automated Evaluation of Comments
in a MOOC Dis...
Content analysis
Image: ALTC2015, Association for Learning Technology ©2015, CC BY-NC-SA 2.0 Automated Evaluation of Comme...
Critical thinking
”...reasonable and reflective thinking that is
focused upon deciding what to do or believe."
S. P. Norri...
Method Categories
Bloom’s
Taxonomy
0
Off-topic
1
Remember
2
Understand
3
Apply
4
Analyse
5
Evaluate
6
Create
Cognitive
Pre...
• 500 comments each MOOC
• Rated according to 2 methods by 7 raters
• Linguistic Inquiry and Word Count (LIWC 2015)
Image:...
Results
Correlation between Bloom and Cognitive Presence
r = 0.909
p = <0.001
Automated Evaluation of Comments
in a MOOC D...
Results
Correlation between CP and Word Count
r = 0.704
p = <0.001
Automated Evaluation of Comments
in a MOOC Discussion F...
0 1 2 3 4
Cognitive Presence
0
10
20
30
40
50
1stpersonpronoun
Sheet 3
Results
Correlation between CP and 1st person singu...
Images: Weka Logo/Weka interface, University of Waikato ©2015 Automated Evaluation of Comments
in a MOOC Discussion Forum
Machine Learning – CP
• Classifiers: Naive Bayes, J48, ZeroR, Random Forest
• 13 attributes:
word count, WPS, Sixltr, ppro...
Predicted
Actual Off-topic Triggering Exploration Integration Resolution
Off-topic 60 0 0 0 0
Triggering 0 58 0 0 12
Explo...
User study
Image: University of Southampton and MOOCAP, Digital Accessibility MOOC.
FutureLearn Ltd 2017
Automated Evaluat...
User study
Image: Automatic rating of all comments in all steps for Week 1, DA MOOC, 2017. Automated Evaluation of Comment...
User study
Image: Automatic rating of all comments in all steps for Week 1.18, DA MOOC,
2017.
Automated Evaluation of Comm...
User study
Image: Automatic rating of all comments in all steps for Week 1.17, DA MOOC,
2017.
Automated Evaluation of Comm...
Initial outcomes
• Questioning importance of monitoring critical thinking
• But...seek reflective, high level comments
• A...
Is critical thinking important?
“I found that it kind of focused on critical thinking, which I
hadn’t really thought about...
Is critical thinking important?
“…in a course like this where it is largely, I suppose partly
informative...and partly des...
Is critical thinking important?
“...depends on to what level of critical thinking you’re trying
to establish, but you need...
Seek reflective comments
“I am actually looking for comments which are reflective in a
sense and maybe come up with extra ...
Value of automatic rating
“I think actually it was about right; it was enough to kind of
get a feel for what was going on ...
Filtering comments
“MOOC’s are always a challenge to be able to look at...even if
it’s not critical thinking you would lik...
Next steps
• Reduce number of categories
• Explore method to include <50 word comments
• Develop a real time application
A...
@studywbv
tim.oriordan@soton.ac.uk
Thank you…
timswww.wordpress.com
Automated Evaluation of Comments
in a MOOC Discussion ...
Upcoming SlideShare
Loading in …5
×

0

Share

Download to read offline

Automated Evaluation of Comments in a MOOC Discussion Forum

Download to read offline

My talk at FutureLearn Academic Network day at CALRG Conference 2017, Open University, Milton Keynes, 16 June 2017.

http://cloudworks.ac.uk/cloudscape/view/3004

  • Be the first to like this

Automated Evaluation of Comments in a MOOC Discussion Forum

  1. 1. Automated Evaluation of Comments in a MOOC Discussion Forum Tim O’Riordan Web and Internet Science (WAIS) www.ecs.soton.ac.uk/people/tor1w07 @studywbv
  2. 2. Motivation Image: Grumpy MOOC cat/ Ryan Tracey, cc-by-nc-nd, © 2014 Automated Evaluation of Comments in a MOOC Discussion Forum
  3. 3. Motivation Image: NodeXL Twitter Search #dataviz/ Marc Smith, cc-by, © 2013 Automated Evaluation of Comments in a MOOC Discussion Forum
  4. 4. Content analysis Image: ALTC2015, Association for Learning Technology ©2015, CC BY-NC-SA 2.0 Automated Evaluation of Comments in a MOOC Discussion Forum
  5. 5. Critical thinking ”...reasonable and reflective thinking that is focused upon deciding what to do or believe." S. P. Norris & R. H. Ennis, Evaluating Critical Thinking. The Practitioners’ Guide to Teaching Thinking Series. Pacific Grove, CA: Midwest Publications, 1989. Automated Evaluation of Comments in a MOOC Discussion Forum
  6. 6. Method Categories Bloom’s Taxonomy 0 Off-topic 1 Remember 2 Understand 3 Apply 4 Analyse 5 Evaluate 6 Create Cognitive Presence (CoI) 0 Off-topic 1 Triggering 2 Exploration 3 Integration 4 Resolution Content Analysis Methods Automated Evaluation of Comments in a MOOC Discussion Forum D. R. Krathwohl, “A Revision of Bloom’s Taxonomy: An Overview,” Theory Pract., vol. 41, no. 4, pp. 212–218, 2002. D. R. Garrison, T. Anderson, and W. Archer, “Critical thinking, cognitive presence, and computer conferencing in distance education,” Am. J. Distance Educ., vol. 15, no. 1, pp. 7–23, 2001.
  7. 7. • 500 comments each MOOC • Rated according to 2 methods by 7 raters • Linguistic Inquiry and Word Count (LIWC 2015) Image: University of Southampton, Understanding Language, Exploring Oceans and Contract Management MOOCs. FutureLearn Ltd 2015. Automated Evaluation of Comments in a MOOC Discussion Forum
  8. 8. Results Correlation between Bloom and Cognitive Presence r = 0.909 p = <0.001 Automated Evaluation of Comments in a MOOC Discussion Forum
  9. 9. Results Correlation between CP and Word Count r = 0.704 p = <0.001 Automated Evaluation of Comments in a MOOC Discussion Forum 0 1 2 3 4 Cognitive Presence 0 50 100 150 200 WordCount Sheet 2
  10. 10. 0 1 2 3 4 Cognitive Presence 0 10 20 30 40 50 1stpersonpronoun Sheet 3 Results Correlation between CP and 1st person singular r = -0.317 p = <0.001 Automated Evaluation of Comments in a MOOC Discussion Forum
  11. 11. Images: Weka Logo/Weka interface, University of Waikato ©2015 Automated Evaluation of Comments in a MOOC Discussion Forum
  12. 12. Machine Learning – CP • Classifiers: Naive Bayes, J48, ZeroR, Random Forest • 13 attributes: word count, WPS, Sixltr, ppron, auxverb, conj, posemo, negemo, cogproc, cause, differ, affiliation, power • <50 word comments • Accuracy: 69% • Cohen’s k: 0.612 Automated Evaluation of Comments in a MOOC Discussion Forum
  13. 13. Predicted Actual Off-topic Triggering Exploration Integration Resolution Off-topic 60 0 0 0 0 Triggering 0 58 0 0 12 Exploration 3 9 28 11 8 Integration 1 3 15 23 19 Resolution 1 2 5 14 38 Confusion matrix for best model – Cognitive presence Machine Learning – CP Automated Evaluation of Comments in a MOOC Discussion Forum
  14. 14. User study Image: University of Southampton and MOOCAP, Digital Accessibility MOOC. FutureLearn Ltd 2017 Automated Evaluation of Comments in a MOOC Discussion Forum
  15. 15. User study Image: Automatic rating of all comments in all steps for Week 1, DA MOOC, 2017. Automated Evaluation of Comments in a MOOC Discussion Forum
  16. 16. User study Image: Automatic rating of all comments in all steps for Week 1.18, DA MOOC, 2017. Automated Evaluation of Comments in a MOOC Discussion Forum
  17. 17. User study Image: Automatic rating of all comments in all steps for Week 1.17, DA MOOC, 2017. Automated Evaluation of Comments in a MOOC Discussion Forum
  18. 18. Initial outcomes • Questioning importance of monitoring critical thinking • But...seek reflective, high level comments • Automatic ratings considered reasonably accurate • Value feedback that facilitates filtering high volumes Automated Evaluation of Comments in a MOOC Discussion Forum
  19. 19. Is critical thinking important? “I found that it kind of focused on critical thinking, which I hadn’t really thought about. And what would be interesting would be to, when you’re designing the course to be thinking in those terms. I suppose when you’re designing a course you’re looking at the getting material across and doing the job or whatever, but you’re not thinking critical thinking as a way of engagement. You’re more worried about what the research says about MOOCs.” Automated Evaluation of Comments in a MOOC Discussion Forum
  20. 20. Is critical thinking important? “…in a course like this where it is largely, I suppose partly informative...and partly designed about quick takeaway, I think it might be a secondary concern…” Automated Evaluation of Comments in a MOOC Discussion Forum
  21. 21. Is critical thinking important? “...depends on to what level of critical thinking you’re trying to establish, but you need to have … some sort of element to show where the learner’s got to.” Automated Evaluation of Comments in a MOOC Discussion Forum
  22. 22. Seek reflective comments “I am actually looking for comments which are reflective in a sense and maybe come up with extra information but also maybe ask questions… these are the best to show that they have learned something.” Automated Evaluation of Comments in a MOOC Discussion Forum
  23. 23. Value of automatic rating “I think actually it was about right; it was enough to kind of get a feel for what was going on and particularly for the visualisation, to actually see it was excellent, it was really helpful.” Automated Evaluation of Comments in a MOOC Discussion Forum
  24. 24. Filtering comments “MOOC’s are always a challenge to be able to look at...even if it’s not critical thinking you would like to look at the comments in a more efficient way... It will save time and it will give a highlight to things that are important.” Automated Evaluation of Comments in a MOOC Discussion Forum
  25. 25. Next steps • Reduce number of categories • Explore method to include <50 word comments • Develop a real time application Automated Evaluation of Comments in a MOOC Discussion Forum
  26. 26. @studywbv tim.oriordan@soton.ac.uk Thank you… timswww.wordpress.com Automated Evaluation of Comments in a MOOC Discussion Forum

My talk at FutureLearn Academic Network day at CALRG Conference 2017, Open University, Milton Keynes, 16 June 2017. http://cloudworks.ac.uk/cloudscape/view/3004

Views

Total views

223

On Slideshare

0

From embeds

0

Number of embeds

2

Actions

Downloads

0

Shares

0

Comments

0

Likes

0

×