Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Learnersourcing: Improving Learning with Collective Learner Activity

1,683 views

Published on

Slides from my thesis defense: "Learnersourcing: Improving Learning with Collective Learner Activity"

Millions of learners today are watching videos on online platforms, such as Khan Academy, YouTube, Coursera, and edX, to take courses and master new skills. But existing video interfaces are not designed to support learning, with limited interactivity and lack of information about learners' engagement and content. Making these improvements requires deep semantic information about video that even state-of-the-art AI techniques cannot fully extract. I take a data-driven approach to address this challenge, using large-scale learning interaction data to dynamically improve video content and interfaces. Specifically, this thesis introduces learnersourcing, a form of crowdsourcing in which learners collectively contribute novel content for future learners while engaging in a meaningful learning experience themselves. I present learnersourcing applications designed for massive open online course videos and how-to tutorial videos, where learners' collective activities 1) highlight points of confusion or importance in a video, 2) extract a solution structure from a tutorial, and 3) improve the navigation experience for future learners. This thesis demonstrates how learnersourcing can enable more interactive, collaborative, and data-driven learning.

Published in: Education

Learnersourcing: Improving Learning with Collective Learner Activity

  1. 1. Juho Kim (MIT CSAIL) Learnersourcing: Improving Learning with Collective Learner Activity
  2. 2. Video learning at scale
  3. 3. Video enables learning at scale
  4. 4. # of learners millionsone hundreds Scalable delivery ≠ Scalable learning
  5. 5. In-person learning: Direct learner-instructor interaction Effective pedagogy
  6. 6. Video learning: Mediated learner-instructor interaction Video interfaces are limiting.
  7. 7. no information about learners Challenges in video learning at scale ? ? no information about content lack of interactivity
  8. 8. Challenges in video learning at scale Understand learners’ engagement Understand video content Support interactive learning
  9. 9. Data-Driven Approach use data from learner interaction to understand and improve learning second-by-second process tracking data-driven content & UI updates
  10. 10. Learnersourcing crowdsourcing with learners as a crowd
  11. 11. Learnersourcing crowdsourcing with learners as a crowd inherently motivated naturally engaged
  12. 12. Learnersourcing crowdsourcing with learners as a crowd Learners’ collective learning activities dynamically improve content & UI for future learners. inherently motivated naturally engaged
  13. 13. Learners watch videos. UI provides social navigation & recommendation. System analyzes interaction traces for hot spots. [Learner3879, Video327, “play”, 35.6] [Learner3879, Video327, “pause”, 47.2] … Video player adapts to collective learner engagement
  14. 14. Learners are prompted to summarize video sections. UI presents a video outline. System coordinates learner tasks for a final summary. What’s the overall goal of the section you just watched? X ……………… V ……………… X ……………… X ……………… Video player coordinates learners to generate a video outline
  15. 15. Two types of learnersourcing Passive track what learners are doing Active ask learners to engage in activities
  16. 16. Learnersourcing applications for educational videos ToolScape [CHI 2014] Interaction Peaks [L@S 2014] LectureScape [UIST 2014] RIMES [CHI 2015] Mudslide [CHI 2015]Crowdy [CSCW 2015]
  17. 17. Learnersourcing requires a multi-disciplinary approach Crowdsourcing – Quality control, Task design, Large-scale input mgmt Social computing – Incentive design, Sense of community among learners UI design – Data-driven & dynamic interaction techniques Video content analysis – Computer vision, Natural language processing Learning science – Pedagogically useful activity, Theoretical background
  18. 18. Thesis statement “In large-scale video learning environments, interfaces powered by learnersourcing can enhance content navigation, create a sense of learning with others, and improve engagement and learning.”
  19. 19. I. Passive learnersourcing (MOOC videos) – Video player clickstream analysis [L@S 2014a, L@S 2014b] – Data-driven content navigation [UIST 2014a, UIST 2014b] II. Active learnersourcing (how-to videos) – Step-by-step information [ CHI 2014] – Summary of steps [CSCW 2015]
  20. 20. I. Passive learnersourcing (MOOC videos) – Video player clickstream analysis [L@S 2014a, L@S 2014b] – Data-driven content navigation [UIST 2014a, UIST 2014b] II. Active learnersourcing (how-to videos) – Step-by-step information [ CHI 2014] – Summary of steps [CSCW 2015]
  21. 21. Video lectures in MOOCs
  22. 22. Classrooms: rich, natural interaction data armgov on Flickr | CC by-nc-saMaria Fleischmann / Worldbank on Flickr | CC by-nc-nd Love Krittaya | public domain unknown author | from pc4all.co.kr
  23. 23. liquidnight on Flickr | CC by-nc-sa
  24. 24. First MOOC-scale video interaction analysis Data Source: 4 edX courses (fall 2012) Domains: computer science, statistics, chemistry Video Events: start, end, play, pause, jump Learners Videos Mean Video Length Processed Video Events 127,839 862 7:46 39.3M
  25. 25. Factors affecting video engagement Shorter videos - significant drop after 6 mins Informal shots over studio production - more personal feel helps Tablet drawing tutorials over slides - continuous visual flow helps How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. Philip J. Guo, Juho Kim, Rob Rubin. Learning at Scale 2014. Metric: session length
  26. 26. How do learners navigate videos? • Watch sequentially • Pause • Re-watch • Skip / Skim • Search
  27. 27. Collective interaction traces video time Learner #1 Learner #2 Learner #3 Learner #4 . . . . . . Learner #7888 Learner #7887
  28. 28. Collective interaction traces into interaction patterns video time interaction events
  29. 29. Interaction peaks Temporal peaks in the number of interaction events, where a significant number of learners show similar interaction patterns video time Understanding In-Video Dropouts and Interaction Peaks in Online Lecture Videos. Juho Kim, Philip J. Guo, Daniel T. Seaton, Piotr Mitros, Krzysztof Z. Gajos, Robert C. Miller. Learning at Scale 2014. ? !interaction events
  30. 30. What causes an interaction peak? Video interaction log data Video content analysis – Visual content (video frames) – Verbal content (transcript)
  31. 31. Observation: Visual / Topical transitions in the video often coincide with a peak.
  32. 32. Returning to content interaction video time # play button clicks
  33. 33. Beginning of new material interaction video time # play button clicks
  34. 34. Data-driven video interaction techniques Use interaction peaks to • draw learners’ attention • support diverse navigational needs • create a sense of learning with others
  35. 35. LectureScape: Lecture video player powered by collective watching data Data-driven interaction techniques for improving navigation of educational videos. Juho Kim, Philip J. Guo, Carrie J. Cai, Shang-Wen (Daniel) Li, Krzysztof Z. Gajos, Robert C. Miller. UIST 2014.
  36. 36. “Where did other learners find confusing / important?” “I want a quick overview of this clip.” “I want to see that previous slide.”
  37. 37. Roller coaster
  38. 38. Phantom cursor • Visual & physical emphasis on interaction peaks • Read wear [Hill et al., 1992], Semantic pointing [Blanch et al., 2004], Pseudo-haptic feedback [Lécuyer et al., 2004]
  39. 39. Visual clip highlights • Interaction data + frame processing
  40. 40. pinning
  41. 41. Pinning: Automatic side-by-side view Pinned slide Video stream
  42. 42. Lab study: 12 edX & on-campus students • LectureScape vs baseline interface • Navigation & learning tasks Visual search “Find a slide where the instructor displays on screen examples of the singleton operation.” Problem search “If the step size in an approximation method decreases, does the code run faster or slower?” Summarization “write down the main points of a video in three minutes.”
  43. 43. Diverse navigation patterns With LectureScape: • more non-linear jumps in navigation • more navigation options - rollercoaster timeline - phantom cursor - highlight summary - pinning “[LectureScape] gives you more options. It personalizes the strategy I can use in the task.”
  44. 44. Interaction data give a sense of “learning together” Interaction peaks matched with participants’ points of “confusion” (8/12) and “importance” (6/12) “It’s not like cold-watching. It feels like watching with other students.” “[interaction data] makes it seem more classroom-y, as in you can compare yourself to how other students are learning and what they need to repeat.”
  45. 45. Summary of passive learnersourcing • Unobtrusive, adaptive use of interaction data • Analysis of MOOC-scale video clickstream data • LectureScape: video player powered by learners’ collective watching behavior • Data-driven interaction techniques for social navigation
  46. 46. I. Passive learnersourcing (MOOC videos) – Video player clickstream analysis [L@S 2014a, L@S 2014b] – Data-driven content navigation [UIST 2014a, UIST 2014b] II. Active learnersourcing (how-to videos) – Step-by-step information [ CHI 2014] – Summary of steps [CSCW 2015]
  47. 47. how-to videos online
  48. 48. Navigating how-to videos is hard find repeat skip
  49. 49. How-to videos contain a step-by-step solution structure Apply gradient map
  50. 50. Completeness & detail of instructions [Eiriksdottir and Catrambone, 2011] Proactive & random access in instructional videos [Zhang et al., 2006] Interactivity: stopping, starting and replaying [Tversky et al., 2002] Subgoals: a group of steps representing task structures [Catrambone, 1994, 1998] Seeing and interacting with solution structure helps learning
  51. 51. Learning with solution structure helps
  52. 52. Learning with solution structure helps
  53. 53. Learning with solution structure helps
  54. 54. Improving how-to video learning Interacting with the solution • UI for solution structure navigation Seeing the solution • Extract steps + subgoals at scale
  55. 55. Improving how-to video learning Interacting with the solution • UI for solution structure navigation Seeing the solution • Extract steps + subgoals at scale
  56. 56. ToolScape: Step-aware video player Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. Juho Kim, Phu Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, & Krzysztof Z. Gajos. CHI 2014. Best of CHI Honorable Mention.
  57. 57. work in progress images parts with no visual progress step labels & links
  58. 58. Study: Photoshop design tasks 12 novice Photoshop users manually annotated videos
  59. 59. Baseline ToolScape
  60. 60. Participants felt more confident about their design skills with ToolScape. – Self-efficacy gain – Four 7-Likert scale questions – Mann-Whitney’s U test (Z=2.06, p<0.05), error bar: standard error 1.4 0 1 2 3 4 5 6 7 ToolScape Baseline 0.13.8 3.8 Before task Self-efficacy gain after task
  61. 61. Participants believed they produced better designs with ToolScape. – Self-rating on designs produced – One 7-Likert scale question – Mann-Whitney’s U test (Z=2.70, p<0.01), error bar: standard error 5.3 3.5 0 1 2 3 4 5 6 7 ToolScape Baseline
  62. 62. Participants actually produced better designs with ToolScape. – External rating on designs – Krippendorff’s alpha = 0.753 – Wilcoxon Signed-rank test (W=317, Z=-2.79, p<0.01, r=0.29) – Error bar: standard error 5.7 7.3 0 2 4 6 8 10 12 ToolScape Baseline Ranking: lower is better
  63. 63. Improving how-to video learning Interacting with the solution • UI for solution structure navigation Seeing the solution • Extract steps + subgoals at scale
  64. 64. Extracting solution structure • Step-by-step information extraction • Subgoal label generation
  65. 65. Goals for annotation method • domain-independent • existing videos • non-expert annotators Learners Crowd workers
  66. 66. Crowd-powered algorithms improvement $0.05 3 votes @ $0.01 … Crowd workflow for complex tasks • Soylent [UIST 2010], CrowdForge [UIST 2011], PlateMate [UIST 2011], Turkomatic [CSCW 2012]
  67. 67. Extracting solution structure • Step-by-step information extraction • Subgoal label generation
  68. 68. 3. Before/after results per each step 1. Step time 2. Step label Desired annotations
  69. 69. Multi-stage annotation workflow When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. Juho Kim, Phu Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, & Krzysztof Z. Gajos. CHI 2014. Best of CHI Honorable Mention.
  70. 70. When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
  71. 71. When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
  72. 72. When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
  73. 73. When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
  74. 74. When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video Output timeline
  75. 75. Evaluation • Generalizable? 75 Photoshop / Cooking / Makeup videos • Accurate? precision and recall against trained annotators’ labels • Non-expert annotators?
  76. 76. Across all domains, ~80% precision and recall Domain Precision Recall Cooking 0.77 0.84 Makeup 0.74 0.77 Photoshop 0.79 0.79 All 0.77 0.81 Precision: % correct labels extracted by crowd Recall: % ground truth labels extracted by crowd
  77. 77. Timing is 2.7 seconds off on average Ground truth: one step every 17.3 seconds 2.7 seconds User 1 User 2 User 3
  78. 78. Extracting solution structure • Step-by-step information extraction • Subgoal label generation
  79. 79. • Requires domain experts and knowledge extraction experts to work together. [Catrambone, 2011] • Insight: the subgoal labeling process is a good exercise for learning! – Reflect on – Explain – Summarize Generating subgoal labels is difficult
  80. 80. Multi-stage learnersourcing workflow Learnersourcing Subgoal Labels for How-to Videos. Sarah Weir, Juho Kim, Krzysztof Z. Gajos, & Robert C. Miller. CSCW 2015.
  81. 81. Stage 1. Generate subgoal labels • Learner: summarize
  82. 82. Stage 2. Evaluate candidate labels • Learner: compare
  83. 83. Stage 3. Proofread subgoal labels • Learner: inspect
  84. 84. Sidebar w/ interactive subgoals & steps
  85. 85. Crowdy evaluation • Does participating in learnersourcing improve learning? • Does the learnersourcing workflow produce good subgoal labels?
  86. 86. Study 1: Pedagogical benefits of learnersourcing • 300 Turkers • Intro stats video • IV: 3 video interfaces (Between-subjects) • DV: – Learning • Pretest + Posttest • Retention test (3-5 days after video watching) – Workload (NASA TLX Test)
  87. 87. Baseline - No prompting - No subgoal shown Expert - No prompting - Subgoal shown Crowdy - Prompting - Subgoal shown
  88. 88. Retention test: Crowdy = Expert > Baseline 1-way ANOVA: F(2, 226)=3.6, p< 0.05, partial η2=0.03 Crowdy vs Baseline: p < 0.05, Cohen’s d = 0.38 Expert vs Baseline: p < 0.05, Cohen’s d = 0.35 Error bar: Standard error 1-way ANOVA: F(2, 226)=4.8, p< 0.01, partial η2=0.04 Crowdy vs Baseline: p < 0.05, Cohen’s d = 0.38 Expert vs Baseline: p < 0.01, Cohen’s d = 0.45 Error bar: Standard error 100  79100  75 100  75
  89. 89. Pretest + Posttest scores were not different across conditions. One-way ANOVA: p > 0.05 Error bar: Standard error
  90. 90. Crowdy didn’t add additional workload. Questions on mental demand, physical demand, temporal demand, performance, effort, and frustration 7-point Likert scale (1: low workload, 7: high workload) One-way ANOVA: p > 0.05 Error bar: Standard error
  91. 91. Study 2: Subgoal labeling quality • ~50 web programming + statistics videos • Classroom + live website deployment • ~1,000 participating users (out of ~2,500 visitors) 922 966 527 Stage 1 subgoals created Stage 2 upvotes Stage 3 upvotes
  92. 92. Analyzed 4 most popular videos 4 external raters compared expert vs learner subgoals Subgoal quality evaluation
  93. 93. Majority of learner-generated subgoals were rated as matching or better than expert-generated ones. Analyzed 4 most popular videos 4 external raters compared expert vs learner subgoals
  94. 94. Interview with learners & creator • Learners • Creator “I was more... attentive to watching, to trying to understand what exactly am I watching.” “Having pop up questions means the viewer has to be paying attention.” the choices “...made me feel as though I was on the same wavelength still.”
  95. 95. Learnersourcing design principles Crowdsourcing simple and concrete task quality control data collection microscopic, focused task cost: money Learnersourcing pedagogically meaningful task incentive design learning + data collection overall contribution visible cost: learners’ time & effort
  96. 96. Summary of active learnersourcing • Techniques for extracting solution structure from existing videos • Video UIs for learning with steps & subgoals • Studies on learning benefits + label quality • Learnersourcing activity design: Engaging & pedagogically meaningful tasks, while byproducts make useful information
  97. 97. Future research agenda
  98. 98. • Richer learner responses Learnersourcing research agenda
  99. 99. • Richer learner responses • Large-scale corpus of annotated videos - Multiple learning paths - Deep search, browsing, recommendation Learnersourcing research agenda
  100. 100. • Richer learner responses • Large-scale corpus of annotated videos - Multiple learning paths - Deep search, browsing, recommendation • Completely learnersourced course - Course created, taught, improved entirely by learners Learnersourcing research agenda
  101. 101. Generalizing learnersourcing community-guided planning, discussion, decision making, collaborative work - Conference planning [UIST 2013, CHI 2014, HCOMP 2013, HCOMP 2014] - Civic engagement [ CHI 2015, CHI 2015 EA]
  102. 102. Learning at scale: Does learning scale? learning benefit per learner # of learners
  103. 103. Learning at scale: Does learning scale? learning benefit per learner # of learners
  104. 104. Learning at scale research: Enable the good parts of in-person learning, at scale learning benefit per learner # of learners
  105. 105. Vision for learnersourcing learning benefit per learner Interactive, collaborative, data-driven online education # of learners
  106. 106. Contributions Learnersourcing: support video learning at scale • UIs – Novel video interfaces & data-driven interaction techniques powered by large-scale learning interaction data • Workflows – Techniques for inferring learner engagement from clickstream data, and extracting semantic information from educational videos • Evaluation studies – Studies measuring pedagogical benefits, resulting data quality, and learners’ qualitative experiences
  107. 107. Learnersourcing: Improving Learning with Collective Learner Activity Juho Kim | MIT CSAIL | juhokim@mit.edu | juhokim.com ToolScape Interaction Peaks LectureScape Crowdy

×