Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs
- 1. Deconstructing Disengagement: Analyzing Learner Subpopulations in Massive Open Online Courses René Chris Emily Kizilcec Piech Schneider
- 2. MOOCs (in this paper) are instructionist + individualised • 6-10 weeks long • 2-3 hours of video lectures/week • autograded assessments with regular deadlines • discussion forum
- 3. Massive Open Online Courses Heterogeneous population: Learners join from anywhere in the world, at any age, for any reason
- 4. Defining Success for Open-Access Learners Assessment scores are problematic: • not comparable across courses • not available for all learners because test-taking is not aligned with learner goals
- 5. Defining Success for Open-Access Learners Completion rates are highly problematic: • numerator = certificate earners, i.e. learners who take assessments • denominator = o total enrolled? overestimate; indicator of interest and not participation o total active? how defined? • ignore plurality of learner intentions • no nuance about subpopulations to help us design interventions or customized course features
- 6. Defining Success for Open-Access Learners Process measures hold promise: • conceptualize learning as an ongoing set of interactions with learning objects and other humans • allow early detection and prediction • indicate points for intervention
- 7. Defining Success for Open-Access Learners Assessment scores Completion rates Process measures
- 8. How to classify learners into meaningful subpopulations?
- 9. Classification Criteria Classification methods for MOOC subpopulations: Universal – valid across multiple courses Theory-driven – reflect the processes of learning Parsimonious – based on small, meaningful feature set Predictive – suggest likely outcomes Dynamic – account for new information over time
- 10. Lens for Analysis • Compare subpopulations • Compare courses
- 11. The Data
- 13. Who took these MOOCs?
- 14. A lot of data!
- 15. Gender skew
- 17. HDI skew
- 18. Clustering
- 19. Sub-populations basis? Engaged Not Engaged
- 20. Engagement Ideal Engagement Time
- 21. Coarse Engagement Labels (T) On Track: Did the weekly assignment on time (B) Behind: Did the weekly assignment, but finished after the due date (A) Auditing: Watched videos but did not do the assignment (O) Out: Did not interact with the course, either through videos or assignments We were able to predict who would take the final AUC = 0.96
- 22. The Aggregate Class A = Auditing O = Out T = On Track B = Behind In this picture Out Is not to scale!
- 23. The Aggregate Class A = Auditing O = Out T = On Track B = Behind 5k In this picture Out Is not to scale!
- 24. The Aggregate Class A = Auditing O = Out T = On Track B = Behind 7k In this picture Out Is not to scale!
- 28. Clustering Methodology There were 21,108 paths in the GS class
- 29. Four Prototypical Trajectories Cluster! (k-means of L1 norm)
- 30. Four Prototypical Trajectories And?
- 31. The Four Prototypical Trajectories
- 32. Prototypical Trajectory 1: Completing
- 33. Prototypical Trajectory 2: Auditing
- 34. Prototypical Trajectory 3: Disengaging
- 35. Prototypical Trajectory 4: Sampling
- 36. Four Prototypical Trajectories Consistent across three courses: Auditing learners watch lectures throughout course, but attempt very few assessments Completing learners attempt majority of assessments offered in course Disengaging learners attempt assessments at beginning of the course, but then sparsely watch lectures or disappear entirely Sampling learners briefly explore course by watching a few videos
- 37. Four Prototypical Trajectories The other courses?
- 39. Four Prototypical Trajectories <suspense>
- 41. Four Prototypical Trajectories Same pattern in all classes
- 42. HS Composition [46k] Disengaging Completing Completing Sampling Auditing
- 43. UG Composition [27k] Disengaging Completing Sampling Auditing
- 44. MS Composition [21k] Disengaging Completing Sampling Auditing
- 45. Validation
- 46. Cluster Validation • Different values of k (split by time) • Including “assignment pass” (95% overlap) • Excluding “behind” (94% overlap) • Silhouette of 0.8 (that’s pretty good) • Pass the common sense test
- 47. High Level Clustering Four Engagement in Prototypical MOOCs Patterns
- 48. Results & Recommendations Comparing Trajectories between Courses
- 49. Overall Experience Au d it in g Co mp let ing Di se HS ng ag ing Sa mp lin g Completing (and Auditing) Au have best experience d it in g Co mp let ing Di se UG ng ag ing Sa mp lin g Identify subpopulations early Au d it in g to customize course features Co mp let ing Di se GS ng ag ing Sa mp lin g 3.0 3.5 4.0 4.5 5.0 Overall Experience
- 50. Discussion Forum Au d it in g Co mp let ing Di se HS ng ag ing Completing learners are Sa mp most active on the forum lin g Au d it in g Co mp let ing Di se UG ng ag Causal relationship? ing Sa mp lin g Au d it in g Co mp let Reputation systems & Di se ing GS ng Social features ag ing Sa mp lin g 0.1 0.51.0 2.0 4.0 7.0 10.0 Average Forum Activity
- 51. Geographical Distribution Trend confirmed by top four participating countries United States, India, Russia, United Kingdom
- 52. Gender Au d it in g Co mp let ing Di se HS ng ag Female Completing learners ing Sa underrepresented in mp lin g advanced courses Au d it in g Co mp let ing Di se UG ng Stereotype threat? ag ing Sa Spencer et al., 1999 mp lin g Au d it in g Co mp let ing Frame assessments to Di se GS ng ag ing minimize stereotype threat Sa mp lin g 2 4 6 8 10 12 14 16 Odds Ratio (Male/Female)
- 54. Future Directions Experiments Collaboration and Peer Effects Interface Customization Targeted Interventions Nuanced Analytics Auditing: MOOC-as-a-resource vs. MOOC-as-a-class Disengaging: Early prediction for intervention Reasons to enroll and trajectories Engagement trajectories for real-time analytics in MOOCs Dashboard visualizations
- 55. Thank you! Stanford Lytics Lab lytics.stanford.edu Office of the Vice Provost for Online Learning Roy Pea, Clifford Nass, Daphne Koller Our LAK reviewers Reference S. Spencer, C. Steele, and D. Quinn. Stereotype threat and women’s math performance. Journal of Experimental Social Psychology, 35(1):4–28, 1999.
- 56. More info? René Kizilcec kizilcec@stanford.edu Chris Piech piech@cs.stanford.edu Emily Schneider elfs@cs.stanford.edu Stanford’s Learning Analytics Group: Lytics Lab lytics.stanford.edu Paper: http://goo.gl/OSX72
Editor's Notes
- Better exp. Than DisengagingIdentification via self-report or algorithmice.g. reducing emphasis on assessments for Auditors—but don’t deprive them of the option
- Positive feedback loopInvestigate causal relationshipStudy effect of RS&SF on participation
- GS-level for illustration
- Anyone who prepared for the test can perform well on it.
- Experiments based on learning scienceNot certificate vs. no certificate