Download to read offline
Uploaded bystudywbv
FLAN Presentation, 7 June 2016
The document discusses a study evaluating comments in MOOCs using two frameworks: Bloom's Taxonomy and the Community of Inquiry model, analyzing 500 comments. It reveals significant correlations between levels of cognitive presence and learning outcomes, suggesting methods to categorize discourse in online learning. Preliminary machine learning approaches were employed to assess comment attributes, achieving moderate accuracy.
FLAN Presentation, 7 June 2016
- 1. Evaluating Comments in MOOCDiscussion Forums Tim O’Riordan Web and Internet Science (WAIS) www.ecs.soton.ac.uk/people/tor1w07 @studywbv
- 2. Motivations Image: Boxing Mail– N.Y. Post Office c. 1910. Library of Congress, NKCR @studywbv 2 Evaluating Comments in MOOC Discussion Forums
- 3. Content analysis Image: ALTC2015,Association for Learning Technology ©2015, CC BY-NC- SA 2.0 @studywbv Evaluating Comments in MOOC Discussion Forums 3
- 4. 4 Bloom score Bloomdescriptor 0 - Off-topic There is written content, but not relevant to the subject under discussion. 1 - Remember Recall of specific learned content, including facts, methods, and theories. Verbs: name, describe, relate, find, list, write, tell. 2 - Understand Perception of meaning and being able to make use of knowledge, without understanding full implications. Verbs: explain, compare, discuss, restate, predict, translate, outline. 3 - Apply Tangible application of learned material in new settings. Verbs: show, complete, use, classify, examine, illustrate, implement, solve. 4 - Analyse Deconstruct learned content into its constituent elements in order to clarify concepts and relationships between ideas. Verbs: explain, compare, contrast, examine, identify, investigate, categorise, differentiate, organise. 5 - Evaluate Assess the significance of material and value in specific settings. Verbs: check, decide, rate, choose, recommend, justify, assess, prioritise, critique. 6 - Create Judge the usefulness of different parts of content, and producing a new arrangement. Verbs: synthesise, invent, plan, compose, construct, design, imagine, generate. A popular and well-respected aid to curriculum development that maps learning to six categories of knowledge acquisition. Bloom’s Taxonomy Evaluating Comments in MOOC Discussion Forums
- 5. 5 CoI score CoIdescriptor 0 - Off-topic There is written content, but not relevant to the subject under discussion. 1 – Triggering event A contribution that exhibits a sense of puzzlement deriving from an issue, dilemma or problem. Includes contributions that present background information, ask questions or move the discussion in a new direction. Verbs: evoke, induce, contradict 2 - Exploration A comment that is seeking a fuller explanation of relevant information. This can include brainstorming, questioning and exchanging information. Contributions are unstructured and may include: unsubstantiated contradictions of previous contributions, different unsupported ideas or themes, personal stories, and descriptions or facts that are not used as evidence. Verbs: inquire, diverge, search 3 - Integration Previously developed ideas are connected. Contributions include: references to previous messages followed by substantiated agreements or disagreements; developing and justifying established themes; cautious hypotheses; combining different sources; providing a tentative solution to an issue. Verb: test, conjecture, check 4 - Resolution New ideas are applied, tested and defended with real world examples. This involves methodically testing hypotheses, critiquing content in a systematic manner, and expressing supported intuition and insight. Verb: commit, settle, confirm A highly cited method that categorises 4 phases of discussion. Cognitive Presence (Community of Inquiry) Evaluating Comments in MOOC Discussion Forums
- 6. Data Image: University ofSouthampton, Understanding Language, Exploring Oceans and Contract Management MOOCs. FutureLearn Ltd 2015 @studywbv 6 Evaluating Comments in MOOC Discussion Forums
- 7. • 500 commentseach MOOC • Rated according to 2 methods by 7 raters • Linguistic Inquiry and Word Count (LIWC 2015) • Correlated by: i) ≥50 word comments ii) Aggregated batches of 10 contiguous comments iii) All individual comments Method @studywbv 7 Evaluating Comments in MOOC Discussion Forums Image: University of Southampton, Understanding Language, Exploring Oceans and Contract Management MOOCs. FutureLearn Ltd 2015.
- 8. Results Correlation between Bloomand Cognitive Presence r = 0.909 p = <0.001 @studywbv 8 Evaluating Comments in MOOC Discussion Forums
- 9. Results Correlation between Bloomand Word Count r = 0.687 p = <0.001 @studywbv 9 Evaluating Comments in MOOC Discussion Forums
- 10. Results Correlation between Bloomand 1st person singular r = -0.321 p = <0.001 @studywbv 10 Evaluating Comments in MOOC Discussion Forums
- 11. Images: Weka Logo/Wekainterface, University of Waikato ©2015 11 Evaluating Comments in MOOC Discussion Forums
- 12. Preliminary ML –CP @studywbv 12 Evaluating Comments in MOOC Discussion Forums • 16 attributes: like, word count, WPS, Sixltr, ppron, prep, auxverb, negate, posemo, negemo, cogproc, cause, differ, affiliation, power • 10 fold Cross-validation • Classifiers: Naive Bayes, J48, Random Forest • Accuracy: 60% • Cohen’s K: 0.48
- 13. Preliminary ML @studywbv 13 Evaluating Commentsin MOOC Discussion Forums Predicted Actual Off-topic Triggering Exploration Integration Resolution Off-topic 135 41 36 6 0 Triggering 41 114 115 18 1 Exploration 13 60 295 92 9 Integration 0 8 120 184 70 Resolution 0 0 9 54 291 Confusion matrix for best model – Cognitive presence
- 14. @studywbv tim.oriordan@soton.ac.uk Thank you… www.slideshare.net/studywbv/flan-presentation-7-june-2016 14 Evaluating Commentsin MOOC Discussion Forums timswww.wordpress.com
Editor's Notes
- #2 Hi, I’m Tim O’Riordan. I’m a second year PhD student and member of the Web and Internet Science group at the University of Southampton and I’m presenting the outputs from a Learning Analytics study I’m currently working on to evaluate comments in MOOC discussion forums. Late last year I ran a content analysis project employing 7 people to rate MOOC comment data from 3 FutureLearn MOOCs using two well-known methods – and looked for correlations between them and with language and interaction data. And I‘ve just started using Machine Learning techniques to see if it’s possible to automate this evaluation effectively.
- #3 Why am I interested in this? We know that finding useful content online can be a hit or miss operation. Finding and evaluating online learning resources is a significant hurdle to overcome. Tagging online resources can help – but Web objects are rarely effectively tagged, and ensuring that tags are sufficient, relevant and kept up to date is a problem. Also, some people don’t know what they have - a significant amount of content on the Web that may have value for supporting informal learning isn’t annotated to emphasises its pedagogical usefulness. Maybe there’s some automatic way that can help? Maybe there’s something in what people write online without special encouragement that can help identify useful from less useful resources, and maybe we can automatically categorize and sort these comments in a way that assists discovery.
- #4 A variety of content analysis methods have been developed to evaluate and categorise computer mediated communication at conferences and within virtual learning environments, but little work has been undertaken to automatically evaluate comments at scale. The approach I initially adopted was to look at the comments people make about things online to find out if what they said provided evidence of pedagogical activity, and if these could be rated in some way. So, I looked at applying a pedagogical approach to content analysis to identify levels of engagement with online objects. It turned out I was not alone in doing this and that many studies have used Bloom and Community of Inquiry methods to explore CMC.
- #5 ‘Bloom’s Taxonomy’ (Bloom et al., 1956), has become a popular and well-respected aid to curriculum development and means of classifying degrees of learning. As amended by Krathwohl (2002), Bloom’s Taxonomy consists of a hierarchy that maps learning to six categories of knowledge acquisition (remember, understand, apply, analyse, evaluate, create ) each indicating the achievement of understanding that is deeper than the preceding category. In content analysis studies Yang et al. (2011) align Bloom’s taxonomy with Henri (1992), a precursor of CoI, in addition, Kember's (1999) association of Bloom’s dimensions with Mezirow's (1991) ‘thoughtful action’ category (e.g. writing), and the utility of mapping word types to Bloom’s levels of cognition (Gibson, Kitto and Willis, 2014) are supportive of the use of the Taxonomy in this study.
- #6 The fundamental structure of CoI is based on the interaction of cognitive presence, social presence, and teaching presence, through which knowledge acquisition takes place within learning communities (Garrison, Anderson and Archer, 2001). As the current study is concerned with identifying evidence of critical thinking associated with learning objects, the main focus is on the categorisation of the cognitive presence dimension which attends to the processes of higher-order thinking within four types of dialogue: triggering, exploration, integration, and resolution. These are mapped to stages of dialogue – starting with a initiating event and concluding with statements that resolve the issues under discussion. “The phases of the practical inquiry model are the idealized logical sequence of the process of critical inquiry and, therefore, must not be seen as immutable.” Are changeable, flexible, variable...
- #7 Comment data from three MOOCs offered on the FutureLearn platform in 2014-15 were analysed. 20 steps per week, 3 – 6 weeks > 41,500 registered learners 15,000 ‘social’ learners 174,500 comments containing 8.5 million words were made available on an anonymised csv file.
- #8 manually coded 1500 comments chosen at random from 3 MOOCs Rated according to Bloom’s levels of learning, and CoI’s measures of meaningful and productive discourse. To avoid confusion, raters were given different sets of comments to rate by each method, and had a 10 day gap between methods. intra-class correlation coefficients were calculated between pairs of coders and provided inter-rater reliability scores of 0.832 for Bloom and 0.818 for CoI.
- #9 Reliability = 0.83 (ICC) When comparing pedagogical scores derived from the two frameworks there is a high correlation score of 0.909 (p<.001), suggesting close association between Bloom’s levels of knowledge acquisition and CoI’s measures of meaningful and productive discourse. While they describe learning activity in different ways they seem to be relatively consistent in measuring its presence and strength. But this begs the question – if levels of knowledge acquisition and stages of meaningful discourse are so so closely associated - what exactly are we measuring? Is it knowledge acquisition or discourse development – or both – or just a general indicator of ‘engagement with understanding’ – some kind of proxy for understanding.
- #10 To provide a couple of examples here’s the linear regression for word count...
- #11 And here it is for 1st person singular. Interestingly – going back to the ‘what are we measuring’ question – these two attributes were identified as significant indicators of what makes a good essay in the mid 60’s - as part of project essay grade. So it may be that simple – but I’m not sure. So what I want to do next is to use the data I’ve gathered to create an automated way of evaluating comments and trying it out on a live MOOC. That is – feeding back automatic ratings to educators who are running a MOOC and see if they find them useful. And I’ve started to explore how machine learning can help.
- #12 I’ve just started with Weka – a free Machine learning work bench.
- #13 Number of attributes – 16 (like, word count, WPS, Sixltr, pronoun, ppron, prep, auxverb, negate, posemo, negemo, cogproc, cause, differ, affiliation, power) Rounded – SMOTE - 10 fold Cross-validation Random Forest, Naive Bayes, J48 Accuracy – 60% Cohen’s K – nominal – 0.48 Not entirely discouraging.
- #14 Future work involves learning more about Machine Learning, finding a willing group of FL instructors, and finding out if this sort of feedback on comments is useful.
- #15 http://www.slideshare.net/studywbv/flan-presentation-7-june-2016