Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Learning analytics to identify exploratory dialogue in online discussions


Published on

Social learning analytics are concerned with the process of knowledge construction as learners build knowledge together in their social and cultural environments. One of the most important tools employed during this process is language. In this presentation we take exploratory dialogue, a joint form of co-reasoning, to be an external indicator that learning is taking place. Using techniques developed within the field of computational linguistics, we build on previous work using cue phrases to identify exploratory dialogue within online discussion.

Published in: Education, Business, Technology
  • Be the first to comment

  • Be the first to like this

Learning analytics to identify exploratory dialogue in online discussions

  1. 1. An Evaluation of LearningAnalytics To Identify ExploratoryDialogue in Online DiscussionsRebecca Ferguson, The Open University, UKZhongyu Wei, The Chinese University of Hong KongYulan He, Aston University, UKSimon Buckingham Shum, The Open University, UK
  2. 2. Discourse analyticsThe ways in which learners engage indialogue indicate how they engage withthe ideas of others, how they relatethose ideas to their understanding andhow they explain their own point of view.• Disputational dialogue• Cumulative dialogue• Exploratory dialogue
  3. 3. Exploratory dialogue Category Indicator Challenge But if, have to respond, my view Critique However, I’m not sure, maybe Discussion of resources Have you read, more links Evaluation Good example, good point Explanation Means that, our goals Explicit reasoning Next step, relates to, that’s why Justification I mean, we learned, we observed Reflections of Agree, here is another, makes the point, perspectives of others take your point, your view
  4. 4. Pilot study: LAK 2011 Time Contribution 2:42 PM I hate talking. :-P My question was whether "gadgets" were just basically widgets and we could embed them in various web sites, like Netvibes, Google Desktop, etc. 2:42 PM Thanks, thats great! I am sure I understood everything, but looks inspiring! 2:43 PM Yes why OU tools not generic tools? 2:43 PM Issues of interoperability 2:43 PM The "new" SocialLearn site looks a lot like a corkboard where you can add various widgets, similar to those existing web start pages. 2:43 PM What if we end up with as many apps/gadgets as we have social networks and then we need a recommender for the apps! 2:43 PM My question was on the definition of the crowd in the wisdom of crowds we acsess in the service model? 2:43 PM there are various different flavours of widget e.g. Google gadgets, W3C widgets etc. SocialLearn has gone for Google gadgets
  5. 5. Computational linguistics Interdisciplinary field that deals with statistical and rule-based modelling of natural language from a computational perspectiveZhongyu Wei Yulan He
  6. 6. Three challenges 1. The annotated dataset is limited 2. Text classification problems are typically topic driven – this is not 3. Nevertheless, both dialogue features and topical features need to be taken into account
  7. 7. Self-training from labelledinstances – a problem Pseudo-label ✓ Exploratory Pseudo-label Exploratory ✓ Pseudo-label ✓ Exploratory Including this Pseudo-label ✗ instance would Exploratory degrade the classifier
  8. 8. Self-trainingfrom labelled features• For each turn in the dialogue, consider each unigram (word), bigram (2 words) and trigram (3 words)• Exploratory or non-exploratory?• Take into account word-association probabilities averaged over many pseudo-labelled examplesPseudo-labelN o n ­exploratory Bigrams ✓ Focusing on To improve labelling, take into account the features gives classification of a number (k) of the nearest a more reliable neighbours of that turn in the dialogue classification
  9. 9. Taking context into accountUnlabelled turn in the dialogue, p1 Nearest neighbour pni1 Pseudo label lni1 Confidence level cni1 Nearest neighbour pni2 Pseudo label lni2Pseudo label for that turn, l1 Confidence level cni2 Pseudo-label Nearest neighbour pni3 Non­ Pseudo label lni3 exploratory Confidence level cni3Confidence value for that label, c1 Let k = 3 (look at 3 nearest0.272727271 neighbours)
  10. 10. Checking against context Pseudo-label based on features is considered correct if support value (s) Pseudo-label ? based on context is high enough Non­ exploratory Support value is calculated by taking into account the pseudo labels and confidence values of k nearest neigbours
  11. 11. Checking the pseudo-labels Pseudo-label N on ­ exploratory Nearest neighbour 1 Pseudo-label N o n ­exploratory ? Confidence level 0.333 s = 0.333 + 0 + 0.666 Pseudo-label Exploratory 3If the support valuefor this pseudo label s = 0.333is greater than R then Nearest neighbour 2this turn in the dialogue Confidence level 0.999can be labelled Pseudo-label Because s < R‘non-exploratory’ N on ­ exploratory this turn in the dialogueLet R = 0.5 should not be labelled Nearest neighbour 3 ‘non-exploratory’ Confidence level 0.666
  12. 12. Cue phrases from pilotAgree Does this suggestAlso Draft EvidenceAlthough Example 94 cue phrasesAlternativeAny research Except •Precise but MisunderstandingAre we Good example •Low recallBecauseBut if Good point Good thing aboutChallenge Have we Used to improveClaimDebate Have you looked at accuracy when Have you readDefinitely Here is another classifyingDependsDifficult How are unannotated datasetDiscussion How canDo we have [...]Do you WhyDoes that mean Your view
  13. 13. DatasetTime Contribution Annotated •Elluminate text chat2:43 Issues of interoperabilityPM •Two-day conference2:43 The "new" SocialLearn site looks a lot like a •2,636 dialogue turnsPM corkboard where you can add various widgets, •Mean word tokens similar to those existing web start pages. per turn: 10.142:43 What if we end up with as many apps/gadgets as UnannotatedPM we have social networks and then we need a recommender for the apps! •Elluminate text chat •Three MOOCs •10,568 dialogue turns2:43 My question was on the definition of the crowd in •Mean word tokensPM the wisdom of crowds we acsess in the service model? per turn: 9.24
  14. 14. Manual coding of data subsetCategory Description Examples includeChallenge A challenge identifies Calling into question something that may be Contradicting wrong and in need of Proposing revision correctionEvaluation An evaluation has a Appraising descriptive quality Assessing JudgingExtension An extension builds on, or Applying idea to a new area provides resources that Increasing range of an idea support, discussion Providing related resourcesReasoning Reasoning is the process of Explaining thinking an idea through Justifying your position Reaching a conclusion
  15. 15. Combining methods• Train initial classifier on annotated dataset• Apply trained classifier to un-annotated data• Use self-learned features to find exploratory dialogue• Use cue-phrase matching to improve accuracy• Take context into account using k-nearest neighbours• Add selected instances to the training dataset• Repeat for five iterations or until less than 0.5% of labels are changed
  16. 16. Evaluation criteriaOn a scale of 0 to 1…AccuracyHow many decisions were correct?Pilot 0.5389 SF+CP+KNN = 0.7924PrecisionHow many ‘exploratory’ turns were actually exploratory?Pilot 0.9523 SF+CP+KNN = 0.8083RecallHow many exploratory turns were classified as exploratory?Pilot 0.4241 SF+CP+KNN = 0.8688F1Weighted average of precision and recallPilot 0.5865 SF+CP+KNN = 0.8331
  17. 17. Varying the value of k k Accuracy Precision Recall F1 1 0.7868 0.8007 0.8666 0.8282 3 0.7924 0.8083 0.8688 0.8331 5 0.7881 0.8005 0.8685 0.8292 7 0.7586 0.7505 0.8640 0.8001 Looking at three nearest neighbours gives best results
  18. 18. Making use of the classifier Each colour block represents 10 turns in the dialogueRed blocks are primarily exploratory, blue blocks primarily non-exploratory
  19. 19. Making use of the classifierExploratory turns in the dialogue The line here is set to highlight anyone who had more than 5/6 of their turns classified as exploratory Analytics like these could be used to provide focused support to learners Total turns in the dialogue
  20. 20. Issues Visual literacy How can we share the maximumExploratory turns in the dialogue amount of information while making these analytics easy to use? Assessment for learning How can we use these analytics to motivate and guide, rather than to discourage? Participatory design Total turns in the dialogue How can we involve learners and teachers in learning discussions around these analytics?
  21. 21. Working in the middle space
  22. 22. Conclusion• We proposed and tested a self-training framework• Found it out-performs alternative methods of detecting exploratory dialogue• Developed an annotated corpus for the development of automatic exploratory dialogue detection• Identified areas for future research• Identified ways of applying this work to support learners and educators
  23. 23. SoLAR Storm webinar Yulan HeSenior Lecturer at the School of Engineering and Appied Science, Aston University, UK