Juho Kim, profile picture
Uploaded byJuho Kim
PPTX, PDF2,907 views

Learnersourcing: Improving Learning with Collective Learner Activity

Juho Kim from MIT CSAIL discusses learnersourcing, a method to enhance video learning through collective learner engagement and data-driven interactions. By analyzing learner interactions, this approach seeks to improve video content and user interfaces, fostering better navigation and a collaborative learning environment. The document highlights applications of learnersourcing in educational videos, detailing both passive and active engagement strategies to boost learning outcomes.

Embed presentation

Downloaded 62 times
1 / 111
2 / 111
3 / 111
4 / 111
5 / 111
6 / 111
7 / 111
8 / 111
9 / 111
10 / 111
11 / 111
12 / 111
13 / 111
14 / 111
15 / 111
16 / 111
17 / 111
18 / 111
19 / 111
20 / 111
21 / 111
22 / 111
23 / 111
24 / 111
25 / 111
26 / 111
27 / 111
28 / 111
29 / 111
30 / 111
31 / 111
32 / 111
33 / 111
34 / 111
35 / 111
36 / 111
37 / 111
38 / 111
39 / 111
40 / 111
41 / 111
42 / 111
43 / 111
44 / 111
45 / 111
46 / 111
47 / 111
Most read
48 / 111
49 / 111
50 / 111
51 / 111
52 / 111
53 / 111
54 / 111
55 / 111
56 / 111
57 / 111
58 / 111
59 / 111
60 / 111
61 / 111
62 / 111
63 / 111
64 / 111
65 / 111
66 / 111
67 / 111
68 / 111
69 / 111
70 / 111
71 / 111
72 / 111
73 / 111
74 / 111
75 / 111
76 / 111
77 / 111
78 / 111
79 / 111
80 / 111
81 / 111
82 / 111
83 / 111
84 / 111
85 / 111
86 / 111
87 / 111
88 / 111
89 / 111
90 / 111
91 / 111
92 / 111
93 / 111
94 / 111
95 / 111
Most read
96 / 111
97 / 111
98 / 111
99 / 111
100 / 111
101 / 111
102 / 111
Most read
103 / 111
104 / 111
105 / 111
106 / 111
107 / 111
108 / 111
109 / 111
110 / 111
111 / 111
Juho Kim (MIT CSAIL) Learnersourcing: Improving Learning with Collective Learner Activity
Video learning at scale
Video enables learning at scale
# of learners millionsone hundreds Scalable delivery ≠ Scalable learning
In-person learning: Direct learner-instructor interaction Effective pedagogy
Video learning: Mediated learner-instructor interaction Video interfaces are limiting.
no information about learners Challenges in video learning at scale ? ? no information about content lack of interactivity
Challenges in video learning at scale Understand learners’ engagement Understand video content Support interactive learning
Data-Driven Approach use data from learner interaction to understand and improve learning second-by-second process tracking data-driven content & UI updates
Learnersourcing crowdsourcing with learners as a crowd
Learnersourcing crowdsourcing with learners as a crowd inherently motivated naturally engaged
Learnersourcing crowdsourcing with learners as a crowd Learners’ collective learning activities dynamically improve content & UI for future learners. inherently motivated naturally engaged
Learners watch videos. UI provides social navigation & recommendation. System analyzes interaction traces for hot spots. [Learner3879, Video327, “play”, 35.6] [Learner3879, Video327, “pause”, 47.2] … Video player adapts to collective learner engagement
Learners are prompted to summarize video sections. UI presents a video outline. System coordinates learner tasks for a final summary. What’s the overall goal of the section you just watched? X ……………… V ……………… X ……………… X ……………… Video player coordinates learners to generate a video outline
Two types of learnersourcing Passive track what learners are doing Active ask learners to engage in activities
Learnersourcing applications for educational videos ToolScape [CHI 2014] Interaction Peaks [L@S 2014] LectureScape [UIST 2014] RIMES [CHI 2015] Mudslide [CHI 2015]Crowdy [CSCW 2015]
Learnersourcing requires a multi-disciplinary approach Crowdsourcing – Quality control, Task design, Large-scale input mgmt Social computing – Incentive design, Sense of community among learners UI design – Data-driven & dynamic interaction techniques Video content analysis – Computer vision, Natural language processing Learning science – Pedagogically useful activity, Theoretical background
Thesis statement “In large-scale video learning environments, interfaces powered by learnersourcing can enhance content navigation, create a sense of learning with others, and improve engagement and learning.”
I. Passive learnersourcing (MOOC videos) – Video player clickstream analysis [L@S 2014a, L@S 2014b] – Data-driven content navigation [UIST 2014a, UIST 2014b] II. Active learnersourcing (how-to videos) – Step-by-step information [ CHI 2014] – Summary of steps [CSCW 2015]
I. Passive learnersourcing (MOOC videos) – Video player clickstream analysis [L@S 2014a, L@S 2014b] – Data-driven content navigation [UIST 2014a, UIST 2014b] II. Active learnersourcing (how-to videos) – Step-by-step information [ CHI 2014] – Summary of steps [CSCW 2015]
Video lectures in MOOCs
Classrooms: rich, natural interaction data armgov on Flickr | CC by-nc-saMaria Fleischmann / Worldbank on Flickr | CC by-nc-nd Love Krittaya | public domain unknown author | from pc4all.co.kr
liquidnight on Flickr | CC by-nc-sa
First MOOC-scale video interaction analysis Data Source: 4 edX courses (fall 2012) Domains: computer science, statistics, chemistry Video Events: start, end, play, pause, jump Learners Videos Mean Video Length Processed Video Events 127,839 862 7:46 39.3M
Factors affecting video engagement Shorter videos - significant drop after 6 mins Informal shots over studio production - more personal feel helps Tablet drawing tutorials over slides - continuous visual flow helps How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. Philip J. Guo, Juho Kim, Rob Rubin. Learning at Scale 2014. Metric: session length
How do learners navigate videos? • Watch sequentially • Pause • Re-watch • Skip / Skim • Search
Collective interaction traces video time Learner #1 Learner #2 Learner #3 Learner #4 . . . . . . Learner #7888 Learner #7887
Collective interaction traces into interaction patterns video time interaction events
Interaction peaks Temporal peaks in the number of interaction events, where a significant number of learners show similar interaction patterns video time Understanding In-Video Dropouts and Interaction Peaks in Online Lecture Videos. Juho Kim, Philip J. Guo, Daniel T. Seaton, Piotr Mitros, Krzysztof Z. Gajos, Robert C. Miller. Learning at Scale 2014. ? !interaction events
What causes an interaction peak? Video interaction log data Video content analysis – Visual content (video frames) – Verbal content (transcript)
Observation: Visual / Topical transitions in the video often coincide with a peak.
Returning to content interaction video time # play button clicks
Beginning of new material interaction video time # play button clicks
Data-driven video interaction techniques Use interaction peaks to • draw learners’ attention • support diverse navigational needs • create a sense of learning with others
LectureScape: Lecture video player powered by collective watching data Data-driven interaction techniques for improving navigation of educational videos. Juho Kim, Philip J. Guo, Carrie J. Cai, Shang-Wen (Daniel) Li, Krzysztof Z. Gajos, Robert C. Miller. UIST 2014.
“Where did other learners find confusing / important?” “I want a quick overview of this clip.” “I want to see that previous slide.”
Roller coaster
Phantom cursor • Visual & physical emphasis on interaction peaks • Read wear [Hill et al., 1992], Semantic pointing [Blanch et al., 2004], Pseudo-haptic feedback [Lécuyer et al., 2004]
Visual clip highlights • Interaction data + frame processing
pinning
Pinning: Automatic side-by-side view Pinned slide Video stream
Lab study: 12 edX & on-campus students • LectureScape vs baseline interface • Navigation & learning tasks Visual search “Find a slide where the instructor displays on screen examples of the singleton operation.” Problem search “If the step size in an approximation method decreases, does the code run faster or slower?” Summarization “write down the main points of a video in three minutes.”
Diverse navigation patterns With LectureScape: • more non-linear jumps in navigation • more navigation options - rollercoaster timeline - phantom cursor - highlight summary - pinning “[LectureScape] gives you more options. It personalizes the strategy I can use in the task.”
Interaction data give a sense of “learning together” Interaction peaks matched with participants’ points of “confusion” (8/12) and “importance” (6/12) “It’s not like cold-watching. It feels like watching with other students.” “[interaction data] makes it seem more classroom-y, as in you can compare yourself to how other students are learning and what they need to repeat.”
Summary of passive learnersourcing • Unobtrusive, adaptive use of interaction data • Analysis of MOOC-scale video clickstream data • LectureScape: video player powered by learners’ collective watching behavior • Data-driven interaction techniques for social navigation
I. Passive learnersourcing (MOOC videos) – Video player clickstream analysis [L@S 2014a, L@S 2014b] – Data-driven content navigation [UIST 2014a, UIST 2014b] II. Active learnersourcing (how-to videos) – Step-by-step information [ CHI 2014] – Summary of steps [CSCW 2015]
how-to videos online
Navigating how-to videos is hard find repeat skip
How-to videos contain a step-by-step solution structure Apply gradient map
Completeness & detail of instructions [Eiriksdottir and Catrambone, 2011] Proactive & random access in instructional videos [Zhang et al., 2006] Interactivity: stopping, starting and replaying [Tversky et al., 2002] Subgoals: a group of steps representing task structures [Catrambone, 1994, 1998] Seeing and interacting with solution structure helps learning
Learning with solution structure helps
Learning with solution structure helps
Learning with solution structure helps
Improving how-to video learning Interacting with the solution • UI for solution structure navigation Seeing the solution • Extract steps + subgoals at scale
Improving how-to video learning Interacting with the solution • UI for solution structure navigation Seeing the solution • Extract steps + subgoals at scale
ToolScape: Step-aware video player Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. Juho Kim, Phu Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, & Krzysztof Z. Gajos. CHI 2014. Best of CHI Honorable Mention.
work in progress images parts with no visual progress step labels & links
Study: Photoshop design tasks 12 novice Photoshop users manually annotated videos
Baseline ToolScape
Participants felt more confident about their design skills with ToolScape. – Self-efficacy gain – Four 7-Likert scale questions – Mann-Whitney’s U test (Z=2.06, p<0.05), error bar: standard error 1.4 0 1 2 3 4 5 6 7 ToolScape Baseline 0.13.8 3.8 Before task Self-efficacy gain after task
Participants believed they produced better designs with ToolScape. – Self-rating on designs produced – One 7-Likert scale question – Mann-Whitney’s U test (Z=2.70, p<0.01), error bar: standard error 5.3 3.5 0 1 2 3 4 5 6 7 ToolScape Baseline
Participants actually produced better designs with ToolScape. – External rating on designs – Krippendorff’s alpha = 0.753 – Wilcoxon Signed-rank test (W=317, Z=-2.79, p<0.01, r=0.29) – Error bar: standard error 5.7 7.3 0 2 4 6 8 10 12 ToolScape Baseline Ranking: lower is better
Improving how-to video learning Interacting with the solution • UI for solution structure navigation Seeing the solution • Extract steps + subgoals at scale
Extracting solution structure • Step-by-step information extraction • Subgoal label generation
Goals for annotation method • domain-independent • existing videos • non-expert annotators Learners Crowd workers
Crowd-powered algorithms improvement $0.05 3 votes @ $0.01 … Crowd workflow for complex tasks • Soylent [UIST 2010], CrowdForge [UIST 2011], PlateMate [UIST 2011], Turkomatic [CSCW 2012]
Extracting solution structure • Step-by-step information extraction • Subgoal label generation
3. Before/after results per each step 1. Step time 2. Step label Desired annotations
Multi-stage annotation workflow When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. Juho Kim, Phu Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, & Krzysztof Z. Gajos. CHI 2014. Best of CHI Honorable Mention.
When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
When & What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video Output timeline
Evaluation • Generalizable? 75 Photoshop / Cooking / Makeup videos • Accurate? precision and recall against trained annotators’ labels • Non-expert annotators?
Across all domains, ~80% precision and recall Domain Precision Recall Cooking 0.77 0.84 Makeup 0.74 0.77 Photoshop 0.79 0.79 All 0.77 0.81 Precision: % correct labels extracted by crowd Recall: % ground truth labels extracted by crowd
Timing is 2.7 seconds off on average Ground truth: one step every 17.3 seconds 2.7 seconds User 1 User 2 User 3
Extracting solution structure • Step-by-step information extraction • Subgoal label generation
• Requires domain experts and knowledge extraction experts to work together. [Catrambone, 2011] • Insight: the subgoal labeling process is a good exercise for learning! – Reflect on – Explain – Summarize Generating subgoal labels is difficult
Multi-stage learnersourcing workflow Learnersourcing Subgoal Labels for How-to Videos. Sarah Weir, Juho Kim, Krzysztof Z. Gajos, & Robert C. Miller. CSCW 2015.
Stage 1. Generate subgoal labels • Learner: summarize
Stage 2. Evaluate candidate labels • Learner: compare
Stage 3. Proofread subgoal labels • Learner: inspect
Sidebar w/ interactive subgoals & steps
Crowdy evaluation • Does participating in learnersourcing improve learning? • Does the learnersourcing workflow produce good subgoal labels?
Study 1: Pedagogical benefits of learnersourcing • 300 Turkers • Intro stats video • IV: 3 video interfaces (Between-subjects) • DV: – Learning • Pretest + Posttest • Retention test (3-5 days after video watching) – Workload (NASA TLX Test)
Baseline - No prompting - No subgoal shown Expert - No prompting - Subgoal shown Crowdy - Prompting - Subgoal shown
Retention test: Crowdy = Expert > Baseline 1-way ANOVA: F(2, 226)=3.6, p< 0.05, partial η2=0.03 Crowdy vs Baseline: p < 0.05, Cohen’s d = 0.38 Expert vs Baseline: p < 0.05, Cohen’s d = 0.35 Error bar: Standard error 1-way ANOVA: F(2, 226)=4.8, p< 0.01, partial η2=0.04 Crowdy vs Baseline: p < 0.05, Cohen’s d = 0.38 Expert vs Baseline: p < 0.01, Cohen’s d = 0.45 Error bar: Standard error 100  79100  75 100  75
Pretest + Posttest scores were not different across conditions. One-way ANOVA: p > 0.05 Error bar: Standard error
Crowdy didn’t add additional workload. Questions on mental demand, physical demand, temporal demand, performance, effort, and frustration 7-point Likert scale (1: low workload, 7: high workload) One-way ANOVA: p > 0.05 Error bar: Standard error
Study 2: Subgoal labeling quality • ~50 web programming + statistics videos • Classroom + live website deployment • ~1,000 participating users (out of ~2,500 visitors) 922 966 527 Stage 1 subgoals created Stage 2 upvotes Stage 3 upvotes
Analyzed 4 most popular videos 4 external raters compared expert vs learner subgoals Subgoal quality evaluation
Majority of learner-generated subgoals were rated as matching or better than expert-generated ones. Analyzed 4 most popular videos 4 external raters compared expert vs learner subgoals
Interview with learners & creator • Learners • Creator “I was more... attentive to watching, to trying to understand what exactly am I watching.” “Having pop up questions means the viewer has to be paying attention.” the choices “...made me feel as though I was on the same wavelength still.”
Learnersourcing design principles Crowdsourcing simple and concrete task quality control data collection microscopic, focused task cost: money Learnersourcing pedagogically meaningful task incentive design learning + data collection overall contribution visible cost: learners’ time & effort
Summary of active learnersourcing • Techniques for extracting solution structure from existing videos • Video UIs for learning with steps & subgoals • Studies on learning benefits + label quality • Learnersourcing activity design: Engaging & pedagogically meaningful tasks, while byproducts make useful information
Future research agenda
• Richer learner responses Learnersourcing research agenda
• Richer learner responses • Large-scale corpus of annotated videos - Multiple learning paths - Deep search, browsing, recommendation Learnersourcing research agenda
• Richer learner responses • Large-scale corpus of annotated videos - Multiple learning paths - Deep search, browsing, recommendation • Completely learnersourced course - Course created, taught, improved entirely by learners Learnersourcing research agenda
Generalizing learnersourcing community-guided planning, discussion, decision making, collaborative work - Conference planning [UIST 2013, CHI 2014, HCOMP 2013, HCOMP 2014] - Civic engagement [ CHI 2015, CHI 2015 EA]
Learning at scale: Does learning scale? learning benefit per learner # of learners
Learning at scale: Does learning scale? learning benefit per learner # of learners
Learning at scale research: Enable the good parts of in-person learning, at scale learning benefit per learner # of learners
Vision for learnersourcing learning benefit per learner Interactive, collaborative, data-driven online education # of learners
Contributions Learnersourcing: support video learning at scale • UIs – Novel video interfaces & data-driven interaction techniques powered by large-scale learning interaction data • Workflows – Techniques for inferring learner engagement from clickstream data, and extracting semantic information from educational videos • Evaluation studies – Studies measuring pedagogical benefits, resulting data quality, and learners’ qualitative experiences
Learnersourcing: Improving Learning with Collective Learner Activity Juho Kim | MIT CSAIL | juhokim@mit.edu | juhokim.com ToolScape Interaction Peaks LectureScape Crowdy

More Related Content

PDF
Ecommerce Website Design Proposal PowerPoint Presentation Slides
bySlideTeam
 
PPTX
Bitbucket
byOkba Mahdjoub
 
PPTX
Intro to Git, GitHub, and BitBucket
byLegal Services National Technology Assistance Project (LSNTAP)
 
PDF
Netflix Recommendations - Beyond the 5 Stars
byXavier Amatriain
 
PDF
An Introduction to Marketing Automation & Mautic
byMautic
 
PPTX
Personalized Page Generation for Browsing Recommendations
byJustin Basilico
 
PPT
Website Usability
byVincci Kwong
 
PDF
Basics in User Experience Design, Information Architecture & Usability
bySebastian Waters
 
Ecommerce Website Design Proposal PowerPoint Presentation Slides
bySlideTeam
 
Bitbucket
byOkba Mahdjoub
 
Intro to Git, GitHub, and BitBucket
byLegal Services National Technology Assistance Project (LSNTAP)
 
Netflix Recommendations - Beyond the 5 Stars
byXavier Amatriain
 
An Introduction to Marketing Automation & Mautic
byMautic
 
Personalized Page Generation for Browsing Recommendations
byJustin Basilico
 
Website Usability
byVincci Kwong
 
Basics in User Experience Design, Information Architecture & Usability
bySebastian Waters
 

What's hot

PPTX
Introduction To Web Accessibility
bySteven Swafford
 
PPTX
Monitoramento Inteligente utilizando o ZABBIX
byLuiz Andrade
 
PDF
The 10 Golden Usability Heuristics (Montreal Girl Geeks September 2014)
byElida Arrizza
 
PDF
Artwork Personalization at Netflix
byJustin Basilico
 
PDF
Space Camp :: Introduction to API Security
byPostman
 
PDF
Metrics Driven Design by Joshua Porter
byAndrew Chen
 
PDF
UX lesson 3: Usability First
byJoan Lumanauw
 
PPTX
Progressive Web Apps and React
byMike Melusky
 
PPTX
A Web for Everyone: Accessibility as a design challenge
byWhitney Quesenbery
 
PPTX
Collaborative Filtering at Spotify
byErik Bernhardsson
 
PPTX
LinkedIn talk at Netflix ML Platform meetup Sep 2019
byFaisal Siddiqi
 
PDF
Wireframing Basics - UX and the Design Process by Amber Vasquez
byMightybytes
 
PDF
Introduzione a Git (ITA - 2017)
byValerio Radice
 
PDF
User Research Interviews: All you need to know
byNorthern User Experience
 
PDF
Tableau Conference 2018: Binging on Data - Enabling Analytics at Netflix
byBlake Irvine
 
PDF
E-commerce Website presentation
byDivyanshuShastri1
 
PPTX
BitBucket presentation
byJonathan Lawerh
 
PDF
Why You Should Be Doing Contract-First API Development
byDevenPhillips
 
PDF
Online School Website Presentation
byParth Sathvara
 
PDF
Opencourrier
byKarim Amane
 
Introduction To Web Accessibility
bySteven Swafford
 
Monitoramento Inteligente utilizando o ZABBIX
byLuiz Andrade
 
The 10 Golden Usability Heuristics (Montreal Girl Geeks September 2014)
byElida Arrizza
 
Artwork Personalization at Netflix
byJustin Basilico
 
Space Camp :: Introduction to API Security
byPostman
 
Metrics Driven Design by Joshua Porter
byAndrew Chen
 
UX lesson 3: Usability First
byJoan Lumanauw
 
Progressive Web Apps and React
byMike Melusky
 
A Web for Everyone: Accessibility as a design challenge
byWhitney Quesenbery
 
Collaborative Filtering at Spotify
byErik Bernhardsson
 
LinkedIn talk at Netflix ML Platform meetup Sep 2019
byFaisal Siddiqi
 
Wireframing Basics - UX and the Design Process by Amber Vasquez
byMightybytes
 
Introduzione a Git (ITA - 2017)
byValerio Radice
 
User Research Interviews: All you need to know
byNorthern User Experience
 
Tableau Conference 2018: Binging on Data - Enabling Analytics at Netflix
byBlake Irvine
 
E-commerce Website presentation
byDivyanshuShastri1
 
BitBucket presentation
byJonathan Lawerh
 
Why You Should Be Doing Contract-First API Development
byDevenPhillips
 
Online School Website Presentation
byParth Sathvara
 
Opencourrier
byKarim Amane
 

Viewers also liked

PDF
Organic Crowdsourcing Systems
byJuho Kim
 
PDF
Ad hoc vs. organised orchestration: A comparative analysis of technology-driv...
byLighton Phiri
 
PDF
Understanding the Effects of Streamlining the Orchestration of Learning Activ...
byLighton Phiri
 
PDF
Ph.D. Research Update: Year#4 Annual Progress and Planned Activities
byLighton Phiri
 
PDF
Presenting your Research at the ECTEL Doctoral Consortium
byChristian Glahn
 
PDF
Efficient Online Evaluation of Big Data Stream Classifiers
byAlbert Bifet
 
PDF
CHI2014 - Crowdsourcing Step-by-Step Information Extraction to Enhance Existi...
byJuho Kim
 
PDF
Moa: Real Time Analytics for Data Streams
byAlbert Bifet
 
PPT
1 Year PhD Presentation
byL. Carlos Freire
 
PPTX
Robust Containers by Eric Brewer
byDocker, Inc.
 
PDF
Streamlined Technology-driven Orchestration: Towards Streamlined Technology-d...
byLighton Phiri
 
PDF
Towards Streamlined Technology-driven orchestration for Effective Teaching
byLighton Phiri
 
PDF
The Hierarchy of Engagement
byGreylock Partners
 
Organic Crowdsourcing Systems
byJuho Kim
 
Ad hoc vs. organised orchestration: A comparative analysis of technology-driv...
byLighton Phiri
 
Understanding the Effects of Streamlining the Orchestration of Learning Activ...
byLighton Phiri
 
Ph.D. Research Update: Year#4 Annual Progress and Planned Activities
byLighton Phiri
 
Presenting your Research at the ECTEL Doctoral Consortium
byChristian Glahn
 
Efficient Online Evaluation of Big Data Stream Classifiers
byAlbert Bifet
 
CHI2014 - Crowdsourcing Step-by-Step Information Extraction to Enhance Existi...
byJuho Kim
 
Moa: Real Time Analytics for Data Streams
byAlbert Bifet
 
1 Year PhD Presentation
byL. Carlos Freire
 
Robust Containers by Eric Brewer
byDocker, Inc.
 
Streamlined Technology-driven Orchestration: Towards Streamlined Technology-d...
byLighton Phiri
 
Towards Streamlined Technology-driven orchestration for Effective Teaching
byLighton Phiri
 
The Hierarchy of Engagement
byGreylock Partners
 

Similar to Learnersourcing: Improving Learning with Collective Learner Activity

PPTX
Understanding In-Video Dropouts and Interaction Peaks in Online Lecture Videos
byJuho Kim
 
PDF
CHI2014 Workshop - Leveraging Video Interaction Data and Content Analysis to ...
byJuho Kim
 
PPTX
Interactive Video for Teaching and Learning
byKristen Sosulski
 
PPT
Using Metrics and Cluster Analysis in Video Based Learning
byAlexandros Kleftodimos
 
PDF
Online Teaching Best Practices
byKristen Sosulski
 
PPT
Judy Stern
byccnmtl
 
PDF
Continuing the Transition Into the Age of Video
byZac Woolfitt
 
PDF
5009a270
byCarlos P. Ceb
 
PPTX
軒銘Icalt.pptx
by軒銘 張
 
PPTX
Lecture capture in your toolkit: building digital media into course design
byClive Young
 
PPTX
Rationalise, Response, Results - Keynote Presentation by Dr. Daniel Tan
byREC:all project
 
PPTX
Maximum Impact with Interactive Video
byAllison Selby
 
PDF
Rebecca eynon learning & interaction in moo cs
byoiisdp
 
PPTX
E learning and ILT
byRACC (Richmond Adult Community College)
 
PPTX
Creating and Using Problem-Based Learning in a TVET Classroom
byLeigh Zeitz
 
PPTX
Echo360 Conf May 2012
byClive Young
 
PPT
Educause 08 Part 2
byguest224058a4
 
PDF
Ain shams actlearn_digital_jtp
byJoan-Tomas Pujola
 
ODP
UniSon Workshop 2006
byssorden
 
PPTX
How can we move beyond recorded lectures?
byClive Young
 
Understanding In-Video Dropouts and Interaction Peaks in Online Lecture Videos
byJuho Kim
 
CHI2014 Workshop - Leveraging Video Interaction Data and Content Analysis to ...
byJuho Kim
 
Interactive Video for Teaching and Learning
byKristen Sosulski
 
Using Metrics and Cluster Analysis in Video Based Learning
byAlexandros Kleftodimos
 
Online Teaching Best Practices
byKristen Sosulski
 
Judy Stern
byccnmtl
 
Continuing the Transition Into the Age of Video
byZac Woolfitt
 
5009a270
byCarlos P. Ceb
 
軒銘Icalt.pptx
by軒銘 張
 
Lecture capture in your toolkit: building digital media into course design
byClive Young
 
Rationalise, Response, Results - Keynote Presentation by Dr. Daniel Tan
byREC:all project
 
Maximum Impact with Interactive Video
byAllison Selby
 
Rebecca eynon learning & interaction in moo cs
byoiisdp
 
E learning and ILT
byRACC (Richmond Adult Community College)
 
Creating and Using Problem-Based Learning in a TVET Classroom
byLeigh Zeitz
 
Echo360 Conf May 2012
byClive Young
 
Educause 08 Part 2
byguest224058a4
 
Ain shams actlearn_digital_jtp
byJoan-Tomas Pujola
 
UniSon Workshop 2006
byssorden
 
How can we move beyond recorded lectures?
byClive Young
 

Recently uploaded

PPTX
AN EXTREMELY BORING GENERAL QUIZ FOR UG.pptx
bySriramG47
 
PPTX
Growth & Development MILESTONES (ADOLESCENTS ).pptx
byAneetaSharma15
 
PPTX
10-12-2025 Francois Staring How can Researchers and Initial Teacher Educators...
byEduSkills OECD
 
PPTX
UNIT 1: COMMUNICATION SKILLS, BARRIERS TO COMMUNICATION, PERSPECTIVES IN COMM...
byPOOJA KARVANDE
 
PDF
Most Imp Chapters & Weightage for Boards 2025.pdf
byTamannaSagar
 
PPTX
How to Configure Push & Pull Rule in Odoo 18 Inventory
byCeline George
 
PPTX
PARENTAL ROUTES OF DRUGS ADMINISTRATION .pptx
byAneetaSharma15
 
PDF
Cultivating Greatness Pune's Best Preschools and Schools.pdf
byWellington College
 
PPTX
Access partner report in Odoo 18.2 _ Odoo 19
byCeline George
 
PDF
The voice of the rain poem class 11th.pdf
byReeta Baral
 
PDF
The Tale of Melon City poem ppt by Sahasra
bybitrasahasra
 
PDF
Multiple Myeloma , definition, etiology, PP, CM , DE and management
byNisha Borsa
 
PPTX
Overview of how to Create a Model in Odoo 18
byCeline George
 
PPTX
TAMIS & TEMS - HOW, WHY and THE STEPS IN PROCTOLOGY
byJohn Thanakumar
 
PPTX
3 G8_Q3_L3_ (Cartoon as Representation in Opinion Editorial Article).pptx
byNorafeSahagun
 
PPTX
The Cell & Cell Cycle-detailed structure and function of organelles.pptx
byAshish Umale
 
PPTX
Details of Epithelial and Connective Tissue.pptx
byAshish Umale
 
PPTX
Introduction-to-Anatomy-and-Physiology.pptx
byAshish Umale
 
PPTX
Planning Assessment for Outcome-based Education
byAdri Jovin
 
PPTX
PURPOSIVE SAMPLING IN EDUCATIONAL RESEARCH RACHITHRA RK.pptx
byssuser047b711
 
AN EXTREMELY BORING GENERAL QUIZ FOR UG.pptx
bySriramG47
 
Growth & Development MILESTONES (ADOLESCENTS ).pptx
byAneetaSharma15
 
10-12-2025 Francois Staring How can Researchers and Initial Teacher Educators...
byEduSkills OECD
 
UNIT 1: COMMUNICATION SKILLS, BARRIERS TO COMMUNICATION, PERSPECTIVES IN COMM...
byPOOJA KARVANDE
 
Most Imp Chapters & Weightage for Boards 2025.pdf
byTamannaSagar
 
How to Configure Push & Pull Rule in Odoo 18 Inventory
byCeline George
 
PARENTAL ROUTES OF DRUGS ADMINISTRATION .pptx
byAneetaSharma15
 
Cultivating Greatness Pune's Best Preschools and Schools.pdf
byWellington College
 
Access partner report in Odoo 18.2 _ Odoo 19
byCeline George
 
The voice of the rain poem class 11th.pdf
byReeta Baral
 
The Tale of Melon City poem ppt by Sahasra
bybitrasahasra
 
Multiple Myeloma , definition, etiology, PP, CM , DE and management
byNisha Borsa
 
Overview of how to Create a Model in Odoo 18
byCeline George
 
TAMIS & TEMS - HOW, WHY and THE STEPS IN PROCTOLOGY
byJohn Thanakumar
 
3 G8_Q3_L3_ (Cartoon as Representation in Opinion Editorial Article).pptx
byNorafeSahagun
 
The Cell & Cell Cycle-detailed structure and function of organelles.pptx
byAshish Umale
 
Details of Epithelial and Connective Tissue.pptx
byAshish Umale
 
Introduction-to-Anatomy-and-Physiology.pptx
byAshish Umale
 
Planning Assessment for Outcome-based Education
byAdri Jovin
 
PURPOSIVE SAMPLING IN EDUCATIONAL RESEARCH RACHITHRA RK.pptx
byssuser047b711
 

Learnersourcing: Improving Learning with Collective Learner Activity

  • 1.
    Juho Kim (MITCSAIL) Learnersourcing: Improving Learning with Collective Learner Activity
  • 2.
  • 3.
  • 4.
    # of learners millionsonehundreds Scalable delivery ≠ Scalable learning
  • 5.
    In-person learning: Direct learner-instructorinteraction Effective pedagogy
  • 6.
    Video learning: Mediated learner-instructorinteraction Video interfaces are limiting.
  • 7.
    no information about learners Challengesin video learning at scale ? ? no information about content lack of interactivity
  • 8.
    Challenges in video learningat scale Understand learners’ engagement Understand video content Support interactive learning
  • 9.
    Data-Driven Approach use datafrom learner interaction to understand and improve learning second-by-second process tracking data-driven content & UI updates
  • 10.
  • 11.
    Learnersourcing crowdsourcing with learnersas a crowd inherently motivated naturally engaged
  • 12.
    Learnersourcing crowdsourcing with learnersas a crowd Learners’ collective learning activities dynamically improve content & UI for future learners. inherently motivated naturally engaged
  • 13.
    Learners watch videos. UIprovides social navigation & recommendation. System analyzes interaction traces for hot spots. [Learner3879, Video327, “play”, 35.6] [Learner3879, Video327, “pause”, 47.2] … Video player adapts to collective learner engagement
  • 14.
    Learners are promptedto summarize video sections. UI presents a video outline. System coordinates learner tasks for a final summary. What’s the overall goal of the section you just watched? X ……………… V ……………… X ……………… X ……………… Video player coordinates learners to generate a video outline
  • 15.
    Two types oflearnersourcing Passive track what learners are doing Active ask learners to engage in activities
  • 16.
    Learnersourcing applications for educationalvideos ToolScape [CHI 2014] Interaction Peaks [L@S 2014] LectureScape [UIST 2014] RIMES [CHI 2015] Mudslide [CHI 2015]Crowdy [CSCW 2015]
  • 17.
    Learnersourcing requires a multi-disciplinaryapproach Crowdsourcing – Quality control, Task design, Large-scale input mgmt Social computing – Incentive design, Sense of community among learners UI design – Data-driven & dynamic interaction techniques Video content analysis – Computer vision, Natural language processing Learning science – Pedagogically useful activity, Theoretical background
  • 18.
    Thesis statement “In large-scalevideo learning environments, interfaces powered by learnersourcing can enhance content navigation, create a sense of learning with others, and improve engagement and learning.”
  • 19.
    I. Passive learnersourcing(MOOC videos) – Video player clickstream analysis [L@S 2014a, L@S 2014b] – Data-driven content navigation [UIST 2014a, UIST 2014b] II. Active learnersourcing (how-to videos) – Step-by-step information [ CHI 2014] – Summary of steps [CSCW 2015]
  • 20.
    I. Passive learnersourcing(MOOC videos) – Video player clickstream analysis [L@S 2014a, L@S 2014b] – Data-driven content navigation [UIST 2014a, UIST 2014b] II. Active learnersourcing (how-to videos) – Step-by-step information [ CHI 2014] – Summary of steps [CSCW 2015]
  • 21.
  • 22.
    Classrooms: rich, naturalinteraction data armgov on Flickr | CC by-nc-saMaria Fleischmann / Worldbank on Flickr | CC by-nc-nd Love Krittaya | public domain unknown author | from pc4all.co.kr
  • 23.
    liquidnight on Flickr| CC by-nc-sa
  • 24.
    First MOOC-scale video interactionanalysis Data Source: 4 edX courses (fall 2012) Domains: computer science, statistics, chemistry Video Events: start, end, play, pause, jump Learners Videos Mean Video Length Processed Video Events 127,839 862 7:46 39.3M
  • 25.
    Factors affecting videoengagement Shorter videos - significant drop after 6 mins Informal shots over studio production - more personal feel helps Tablet drawing tutorials over slides - continuous visual flow helps How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. Philip J. Guo, Juho Kim, Rob Rubin. Learning at Scale 2014. Metric: session length
  • 26.
    How do learnersnavigate videos? • Watch sequentially • Pause • Re-watch • Skip / Skim • Search
  • 27.
    Collective interaction traces videotime Learner #1 Learner #2 Learner #3 Learner #4 . . . . . . Learner #7888 Learner #7887
  • 28.
    Collective interaction traces intointeraction patterns video time interaction events
  • 29.
    Interaction peaks Temporal peaksin the number of interaction events, where a significant number of learners show similar interaction patterns video time Understanding In-Video Dropouts and Interaction Peaks in Online Lecture Videos. Juho Kim, Philip J. Guo, Daniel T. Seaton, Piotr Mitros, Krzysztof Z. Gajos, Robert C. Miller. Learning at Scale 2014. ? !interaction events
  • 30.
    What causes aninteraction peak? Video interaction log data Video content analysis – Visual content (video frames) – Verbal content (transcript)
  • 31.
    Observation: Visual /Topical transitions in the video often coincide with a peak.
  • 32.
    Returning to content interaction videotime # play button clicks
  • 33.
    Beginning of newmaterial interaction video time # play button clicks
  • 34.
    Data-driven video interaction techniques Useinteraction peaks to • draw learners’ attention • support diverse navigational needs • create a sense of learning with others
  • 35.
    LectureScape: Lecture videoplayer powered by collective watching data Data-driven interaction techniques for improving navigation of educational videos. Juho Kim, Philip J. Guo, Carrie J. Cai, Shang-Wen (Daniel) Li, Krzysztof Z. Gajos, Robert C. Miller. UIST 2014.
  • 36.
    “Where did otherlearners find confusing / important?” “I want a quick overview of this clip.” “I want to see that previous slide.”
  • 37.
  • 38.
    Phantom cursor • Visual& physical emphasis on interaction peaks • Read wear [Hill et al., 1992], Semantic pointing [Blanch et al., 2004], Pseudo-haptic feedback [Lécuyer et al., 2004]
  • 39.
    Visual clip highlights •Interaction data + frame processing
  • 40.
  • 41.
  • 42.
    Lab study: 12 edX& on-campus students • LectureScape vs baseline interface • Navigation & learning tasks Visual search “Find a slide where the instructor displays on screen examples of the singleton operation.” Problem search “If the step size in an approximation method decreases, does the code run faster or slower?” Summarization “write down the main points of a video in three minutes.”
  • 43.
    Diverse navigation patterns WithLectureScape: • more non-linear jumps in navigation • more navigation options - rollercoaster timeline - phantom cursor - highlight summary - pinning “[LectureScape] gives you more options. It personalizes the strategy I can use in the task.”
  • 44.
    Interaction data give asense of “learning together” Interaction peaks matched with participants’ points of “confusion” (8/12) and “importance” (6/12) “It’s not like cold-watching. It feels like watching with other students.” “[interaction data] makes it seem more classroom-y, as in you can compare yourself to how other students are learning and what they need to repeat.”
  • 46.
    Summary of passivelearnersourcing • Unobtrusive, adaptive use of interaction data • Analysis of MOOC-scale video clickstream data • LectureScape: video player powered by learners’ collective watching behavior • Data-driven interaction techniques for social navigation
  • 47.
    I. Passive learnersourcing(MOOC videos) – Video player clickstream analysis [L@S 2014a, L@S 2014b] – Data-driven content navigation [UIST 2014a, UIST 2014b] II. Active learnersourcing (how-to videos) – Step-by-step information [ CHI 2014] – Summary of steps [CSCW 2015]
  • 48.
  • 49.
    Navigating how-to videosis hard find repeat skip
  • 50.
    How-to videos contain astep-by-step solution structure Apply gradient map
  • 51.
    Completeness & detailof instructions [Eiriksdottir and Catrambone, 2011] Proactive & random access in instructional videos [Zhang et al., 2006] Interactivity: stopping, starting and replaying [Tversky et al., 2002] Subgoals: a group of steps representing task structures [Catrambone, 1994, 1998] Seeing and interacting with solution structure helps learning
  • 52.
    Learning with solutionstructure helps
  • 53.
    Learning with solutionstructure helps
  • 54.
    Learning with solutionstructure helps
  • 55.
    Improving how-to videolearning Interacting with the solution • UI for solution structure navigation Seeing the solution • Extract steps + subgoals at scale
  • 56.
    Improving how-to videolearning Interacting with the solution • UI for solution structure navigation Seeing the solution • Extract steps + subgoals at scale
  • 57.
    ToolScape: Step-aware videoplayer Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. Juho Kim, Phu Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, & Krzysztof Z. Gajos. CHI 2014. Best of CHI Honorable Mention.
  • 58.
    work in progress images partswith no visual progress step labels & links
  • 59.
    Study: Photoshop designtasks 12 novice Photoshop users manually annotated videos
  • 60.
  • 61.
    Participants felt moreconfident about their design skills with ToolScape. – Self-efficacy gain – Four 7-Likert scale questions – Mann-Whitney’s U test (Z=2.06, p<0.05), error bar: standard error 1.4 0 1 2 3 4 5 6 7 ToolScape Baseline 0.13.8 3.8 Before task Self-efficacy gain after task
  • 62.
    Participants believed theyproduced better designs with ToolScape. – Self-rating on designs produced – One 7-Likert scale question – Mann-Whitney’s U test (Z=2.70, p<0.01), error bar: standard error 5.3 3.5 0 1 2 3 4 5 6 7 ToolScape Baseline
  • 63.
    Participants actually produced betterdesigns with ToolScape. – External rating on designs – Krippendorff’s alpha = 0.753 – Wilcoxon Signed-rank test (W=317, Z=-2.79, p<0.01, r=0.29) – Error bar: standard error 5.7 7.3 0 2 4 6 8 10 12 ToolScape Baseline Ranking: lower is better
  • 64.
    Improving how-to videolearning Interacting with the solution • UI for solution structure navigation Seeing the solution • Extract steps + subgoals at scale
  • 65.
    Extracting solution structure •Step-by-step information extraction • Subgoal label generation
  • 66.
    Goals for annotationmethod • domain-independent • existing videos • non-expert annotators Learners Crowd workers
  • 67.
    Crowd-powered algorithms improvement $0.053 votes @ $0.01 … Crowd workflow for complex tasks • Soylent [UIST 2010], CrowdForge [UIST 2011], PlateMate [UIST 2011], Turkomatic [CSCW 2012]
  • 68.
    Extracting solution structure •Step-by-step information extraction • Subgoal label generation
  • 69.
    3. Before/after results pereach step 1. Step time 2. Step label Desired annotations
  • 70.
    Multi-stage annotation workflow When& What are the steps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. Juho Kim, Phu Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, & Krzysztof Z. Gajos. CHI 2014. Best of CHI Honorable Mention.
  • 71.
    When & What are thesteps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
  • 72.
    When & What are thesteps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
  • 73.
    When & What are thesteps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
  • 74.
    When & What are thesteps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video
  • 75.
    When & What are thesteps? Vote & Improve Before/After the steps? FIND VERIFY EXPAND Input video Output timeline
  • 76.
    Evaluation • Generalizable? 75 Photoshop/ Cooking / Makeup videos • Accurate? precision and recall against trained annotators’ labels • Non-expert annotators?
  • 77.
    Across all domains, ~80%precision and recall Domain Precision Recall Cooking 0.77 0.84 Makeup 0.74 0.77 Photoshop 0.79 0.79 All 0.77 0.81 Precision: % correct labels extracted by crowd Recall: % ground truth labels extracted by crowd
  • 78.
    Timing is 2.7seconds off on average Ground truth: one step every 17.3 seconds 2.7 seconds User 1 User 2 User 3
  • 79.
    Extracting solution structure •Step-by-step information extraction • Subgoal label generation
  • 80.
    • Requires domainexperts and knowledge extraction experts to work together. [Catrambone, 2011] • Insight: the subgoal labeling process is a good exercise for learning! – Reflect on – Explain – Summarize Generating subgoal labels is difficult
  • 83.
    Multi-stage learnersourcing workflow Learnersourcing SubgoalLabels for How-to Videos. Sarah Weir, Juho Kim, Krzysztof Z. Gajos, & Robert C. Miller. CSCW 2015.
  • 84.
    Stage 1. Generatesubgoal labels • Learner: summarize
  • 85.
    Stage 2. Evaluatecandidate labels • Learner: compare
  • 86.
    Stage 3. Proofreadsubgoal labels • Learner: inspect
  • 88.
  • 89.
    Crowdy evaluation • Doesparticipating in learnersourcing improve learning? • Does the learnersourcing workflow produce good subgoal labels?
  • 90.
    Study 1: Pedagogicalbenefits of learnersourcing • 300 Turkers • Intro stats video • IV: 3 video interfaces (Between-subjects) • DV: – Learning • Pretest + Posttest • Retention test (3-5 days after video watching) – Workload (NASA TLX Test)
  • 91.
    Baseline - No prompting -No subgoal shown Expert - No prompting - Subgoal shown Crowdy - Prompting - Subgoal shown
  • 92.
    Retention test: Crowdy =Expert > Baseline 1-way ANOVA: F(2, 226)=3.6, p< 0.05, partial η2=0.03 Crowdy vs Baseline: p < 0.05, Cohen’s d = 0.38 Expert vs Baseline: p < 0.05, Cohen’s d = 0.35 Error bar: Standard error 1-way ANOVA: F(2, 226)=4.8, p< 0.01, partial η2=0.04 Crowdy vs Baseline: p < 0.05, Cohen’s d = 0.38 Expert vs Baseline: p < 0.01, Cohen’s d = 0.45 Error bar: Standard error 100  79100  75 100  75
  • 93.
    Pretest + Posttestscores were not different across conditions. One-way ANOVA: p > 0.05 Error bar: Standard error
  • 94.
    Crowdy didn’t add additionalworkload. Questions on mental demand, physical demand, temporal demand, performance, effort, and frustration 7-point Likert scale (1: low workload, 7: high workload) One-way ANOVA: p > 0.05 Error bar: Standard error
  • 95.
    Study 2: Subgoallabeling quality • ~50 web programming + statistics videos • Classroom + live website deployment • ~1,000 participating users (out of ~2,500 visitors) 922 966 527 Stage 1 subgoals created Stage 2 upvotes Stage 3 upvotes
  • 96.
    Analyzed 4 mostpopular videos 4 external raters compared expert vs learner subgoals Subgoal quality evaluation
  • 97.
    Majority of learner-generatedsubgoals were rated as matching or better than expert-generated ones. Analyzed 4 most popular videos 4 external raters compared expert vs learner subgoals
  • 98.
    Interview with learners& creator • Learners • Creator “I was more... attentive to watching, to trying to understand what exactly am I watching.” “Having pop up questions means the viewer has to be paying attention.” the choices “...made me feel as though I was on the same wavelength still.”
  • 99.
    Learnersourcing design principles Crowdsourcing simple andconcrete task quality control data collection microscopic, focused task cost: money Learnersourcing pedagogically meaningful task incentive design learning + data collection overall contribution visible cost: learners’ time & effort
  • 100.
    Summary of activelearnersourcing • Techniques for extracting solution structure from existing videos • Video UIs for learning with steps & subgoals • Studies on learning benefits + label quality • Learnersourcing activity design: Engaging & pedagogically meaningful tasks, while byproducts make useful information
  • 101.
  • 102.
    • Richer learnerresponses Learnersourcing research agenda
  • 103.
    • Richer learnerresponses • Large-scale corpus of annotated videos - Multiple learning paths - Deep search, browsing, recommendation Learnersourcing research agenda
  • 104.
    • Richer learnerresponses • Large-scale corpus of annotated videos - Multiple learning paths - Deep search, browsing, recommendation • Completely learnersourced course - Course created, taught, improved entirely by learners Learnersourcing research agenda
  • 105.
    Generalizing learnersourcing community-guided planning,discussion, decision making, collaborative work - Conference planning [UIST 2013, CHI 2014, HCOMP 2013, HCOMP 2014] - Civic engagement [ CHI 2015, CHI 2015 EA]
  • 106.
    Learning at scale: Doeslearning scale? learning benefit per learner # of learners
  • 107.
    Learning at scale: Doeslearning scale? learning benefit per learner # of learners
  • 108.
    Learning at scaleresearch: Enable the good parts of in-person learning, at scale learning benefit per learner # of learners
  • 109.
    Vision for learnersourcing learning benefit per learner Interactive,collaborative, data-driven online education # of learners
  • 110.
    Contributions Learnersourcing: support videolearning at scale • UIs – Novel video interfaces & data-driven interaction techniques powered by large-scale learning interaction data • Workflows – Techniques for inferring learner engagement from clickstream data, and extracting semantic information from educational videos • Evaluation studies – Studies measuring pedagogical benefits, resulting data quality, and learners’ qualitative experiences
  • 111.
    Learnersourcing: Improving Learningwith Collective Learner Activity Juho Kim | MIT CSAIL | juhokim@mit.edu | juhokim.com ToolScape Interaction Peaks LectureScape Crowdy

Editor's Notes

  • #3 Video has emerged as a primary medium for online learning, and millions of learners today are watching videos on web platforms such as Khan Academy, Coursera, edX, and YouTube.
  • #4 Videos are great in that they can be accessed any time from anywhere, and learners can watch at their own pace. Also, once a video is published, millions of learners can learn from the same material. It’s an efficient and scalable way to deliver education.
  • #5 [1 min] But does it necessarily mean that video is a better medium for learning? Unfortunately, I would argue that the answer is no, at least not at the moment. Let’s compare video learning against some of its competitors.
  • #6 In in-person learning, like 1:1 private tutoring or classroom lectures, the learner and the instructor interact directly. This direct link enables many effective instructional strategies, So that learners can stay engaged, while instructors can provide immediate feedback and adapt their instruction.
  • #7 In video learning, however, there is no direct channel between the learner and the instructor. The video interface is in the middle. While this asynchronous interaction makes scalable delivery possible, There are trade-offs involved. Many of the good ingredients in 1:1 learning are missing. // most video interfaces are not really designed to support learning. As a result, many video learners resort to passive and isolated viewing. And the video interface doesn’t really adapt to the learner, Finding information and navigating to specific points inside a video is also difficult. “video search has been a notoriously difficult problem.” I think these limitations come from two major problems that most video interfaces have. First, video interfaces don’t know who the learners are and how they are watching the video. Second of all, video interfaces don’t know what the content is and how to make it more helpful for learners. === Human computation involves computational problems      - Replaces computers with humans      - Hard for computers, easy for humans. Or they work together to make it easy Crowdsourcing      - Replaces human workers with members of the public      - ”Outsourcing it to an undefined, generally large group of people in the form of an open call" - Jeff Howe Human computation is a different perspective on how humans and computers interact. It is different in that the human participation is determined by the computational framework. Humans are helping computers in the computation, and crowd intelligence is augmenting the computational power that the computer has.
  • #8 For video learning to truly scale, I think it’s essential that the video interfaces understand the learners’ engagement and understand the content better. Understanding learner engagement is hard because the interfaces cannot reach beyond the screen to really know what learners are doing. “out in the wild” Understanding content is hard because it requires understanding the domain, learning objective, and presentational approach, Each of which is a challenging problem on its own. “out in the wild” === Yet we hope to be able to analyze learners’ video usage patterns and watching behaviors. (Interaction pattern & behavior analysis) And we hope to be able to mine, index, and summarize the video. (Video mining, indexing, summarization)
  • #9 For video learning to truly scale, I think it’s essential that the video interfaces understand the learners’ engagement and understand the content better. Understanding learner engagement is hard because the interfaces cannot reach beyond the screen to really know what learners are doing. “out in the wild” Understanding content is hard because it requires understanding the domain, learning objective, and presentational approach, Each of which is a challenging problem on its own. “out in the wild” === Yet we hope to be able to analyze learners’ video usage patterns and watching behaviors. (Interaction pattern & behavior analysis) And we hope to be able to mine, index, and summarize the video. (Video mining, indexing, summarization)
  • #10 For video learning to truly scale, I think it’s essential that the video interfaces understand the learners’ engagement and understand the content better. Understanding learner engagement is hard because the interfaces cannot reach beyond the screen to really know what learners are doing. “out in the wild” Understanding content is hard because it requires understanding the domain, learning objective, and presentational approach, Each of which is a challenging problem on its own. “out in the wild” === Yet we hope to be able to analyze learners’ video usage patterns and watching behaviors. (Interaction pattern & behavior analysis) And we hope to be able to mine, index, and summarize the video. (Video mining, indexing, summarization)
  • #11 For video learning to truly scale, I think it’s essential that the video interfaces understand the learners’ engagement and understand the content better. Understanding learner engagement is hard because the interfaces cannot reach beyond the screen to really know what learners are doing. “out in the wild” Understanding content is hard because it requires understanding the domain, learning objective, and presentational approach, Each of which is a challenging problem on its own. “out in the wild” === Yet we hope to be able to analyze learners’ video usage patterns and watching behaviors. (Interaction pattern & behavior analysis) And we hope to be able to mine, index, and summarize the video. (Video mining, indexing, summarization)
  • #12 To address these challenges in video learning, I take a data-driven approach in my research. I use data generated from learners’ interaction with video content to understand and improve learning. The fact that we have a large group of learners watching the same video provides some unique opportunities. We now have a way to track the learning process at a fine-grained level: second by second and click by click. And use this understanding to improve the content and video interfaces. What becomes important here are the tools to collect, process, and present this large-scale learning interaction data.
  • #13 too much focus on what learners DO. I’m telling my trick already. There’s no “wow, this is a neat solution to the problem”, because I’m giving away the suspense too easily. When I present my solution, people should be reacting, “oh, wow, that’s a neat solution" === And my secret recipe is learnersourcing. As the name implies, learnersourcing is crowdsourcing with learners as a crowd. As we know, crowdsourcing has offered a new solution to many computationally difficult problems by introducing human intelligence as a building block. While crowdsourcing often issues an open call to an undefined crowd, learnersourcing uses a specialized crowd, learners who are inherently motivated and naturally engaged in their learning. This difference makes learnersourcing tackle unique problems that computers or general crowdsourcing cannot easily solve. And the idea is that the byproduct of learners’ natural learning activities can be used to Dynamically improve the future learner’s experience. === Natural interaction with content, Or design special interactions to collect useful input from learners
  • #14 too much focus on what learners DO. I’m telling my trick already. There’s no “wow, this is a neat solution to the problem”, because I’m giving away the suspense too easily. When I present my solution, people should be reacting, “oh, wow, that’s a neat solution" === And my secret recipe is learnersourcing. As the name implies, learnersourcing is crowdsourcing with learners as a crowd. As we know, crowdsourcing has offered a new solution to many computationally difficult problems by introducing human intelligence as a building block. While crowdsourcing often issues an open call to an undefined crowd, learnersourcing uses a specialized crowd, learners who are inherently motivated and naturally engaged in their learning. This difference makes learnersourcing tackle unique problems that computers or general crowdsourcing cannot easily solve. And the idea is that the byproduct of learners’ natural learning activities can be used to Dynamically improve the future learner’s experience. === Natural interaction with content, Or design special interactions to collect useful input from learners
  • #15 too much focus on what learners DO. I’m telling my trick already. There’s no “wow, this is a neat solution to the problem”, because I’m giving away the suspense too easily. When I present my solution, people should be reacting, “oh, wow, that’s a neat solution" === And my secret recipe is learnersourcing. As the name implies, learnersourcing is crowdsourcing with learners as a crowd. As we know, crowdsourcing has offered a new solution to many computationally difficult problems by introducing human intelligence as a building block. While crowdsourcing often issues an open call to an undefined crowd, learnersourcing uses a specialized crowd, learners who are inherently motivated and naturally engaged in their learning. This difference makes learnersourcing tackle unique problems that computers or general crowdsourcing cannot easily solve. And the idea is that the byproduct of learners’ natural learning activities can be used to Dynamically improve the future learner’s experience. === Natural interaction with content, Or design special interactions to collect useful input from learners
  • #16 [5 min] Let me give you some concrete examples of what I mean by learnersourcing. A 30-second tour. What if the video understood how learners watch the video and adapt? Here’s a video player that adapts to learners’ watching behavior. As the learner watches a video, the clickstream log is analyzed by the system for a meaningful pattern. The system finds hot spots in the video where many learners were feeling confused about. Based on this information, the video interface dynamically improves various features for future learners.
  • #17 I implemented this idea in a video player called LectureScape. I’ll talk about it in more detail later in the talk.
  • #18 Let’s look at another example. This time, the video player adds a new learning activity for video viewers. Learners are asked to summarize the section that they just watched. The system collects summary labels from multiple learners and finalized into a video outline. The UI then adds this outline next to the video player for future learners.
  • #19 Some colleagues and I implemented this idea in a video player called Crowdy. I’ll also talk about it in more detail later in the talk.
  • #20 I categorize learnersourcing into two models. In passive learning, I use existing data streams coming from leaners’ video watching activity to process and analyze them into meaningful information for future learners. On the other hand, in active learning, I design new learning activities that benefit learners AND contribute data that can be used to improve learning exp. for the student.
  • #21 In my research, I’ve designed, built, and studied various active and passive learnersourcing applications to support video learning at scale. I’ll cover most of them at least briefly later.
  • #23 [7 min] Essentially, in learnersourcing, I strive to establish a feedback loop between the learner and the system. As the learner watches the video naturally or answers some prompt that is pedagogically meaningful, The byproduct of these activities are processed by the system to produce a meaningful outcome. Then the system used this outcome to dynamically improve content and UI for future learners. Hopefully by the end of the talk, you’ll be convinced that learnersourcing can… In large-scale video learning environments, interfaces powered by learnersourcing can enhance content navigation, create a sense of learning with others, and ultimately improve learning.
  • #24 [7 min] Essentially, in learnersourcing, I strive to establish a feedback loop between the learner and the system. As the learner watches the video naturally or answers some prompt that is pedagogically meaningful, The byproduct of these activities are processed by the system to produce a meaningful outcome. Then the system used this outcome to dynamically improve content and UI for future learners. Hopefully by the end of the talk, you’ll be convinced that learnersourcing can… In large-scale video learning environments, interfaces powered by learnersourcing can enhance content navigation, create a sense of learning with others, and ultimately improve learning.
  • #27 There are more than 6000 MOOCs now, some of which are from this very school and some from faculty in this room. A MOOC often include tens to hundreds of video clips, and research shows that students taking a MOOC spend a majority of their time watching videos. But MOOC instructors often don’t have a good sense of how learners are using the videos.
  • #28 Traditional classrooms, on the other hand, provide natural interaction data. Instructors in a classroom can visually check students’ engagement. Students might be engaged and paying attention, confused with a question, or bored and falling asleep. Or it might be the entire class that falls asleep. === relate to the current room and how I can adapt dynamically While online videos provide access any time from anywhere, they disconnect the interaction channel between instructor and students. One problem of videos is that interaction between student and instructor, and student and student is === http://www.flickr.com/photos/10816734@N03/8249942979/in/photolist-dz27Cc-7FGT5v-fvhqZF-auPupW-8c6a4M-9uEhUU-dZUzSs-i54S8p-i55y7F-i54SgR-i54ZH3-i55xKi-i55179-i55yck-i54S3V-i55y6i-i54ZLE-i55xY4-i553f9-cRANHJ-98HXSU-apaQNA-aGNKak-cJ2nes-cJ2niN-cJ2nuh-cJ2nbq-cJ2npU-8aJrP5-dgF74D-bBLYjg-ddYkR6-9qqiYV-98HTZN-bEhdMJ-7WnzsU-88biBa-8qU4Cy-eEknDg-7Cpa5o-eEoW74-dmhvoT-9f7aPF-9irnJd-brfBxH-bpxmdx-87vrbw-8tXMNr-am1uZL-cXJdEo-bDpXMx http://www.flickr.com/photos/armgov/4991079510/sizes/z/ https://www.pc4all.co.kr/new_co_upload_view.asp?num=448&category=comm
  • #29 For instructors recording videos for MOOCs, it’s like they are talking to a wall. Instructors have no way to see if students are engaged, confused, or bored. Students also have no way to express their reaction, or see how other students are watching the video. This seriously restricts our understanding of how students learn with the videos, and limits the video learning experience itself. === // But a MOOC video experience is not like a classroom experience. // There’s a disconnect between instructors and students, in both time and location. http://www.flickr.com/photos/liquidnight/2214264870/sizes/l/
  • #31 We first looked at a few major factors that might affect learners’ engagement with video. By performing a post-hoc analysis looking at average session length as a metric, We found that … These are useful guidelines for instructors and video editors, and when we interviewed a video editor at edX, He enjoyed the fact that much of what he knew from his experience and intuition has been confirmed. We wanted to now look at how learners navigate inside a video.
  • #32 Watch sequentially: common when you watch for the first time Pause: you might be confused, pace is too fast Return: revisit important concept Skip: search for a specific part, or quickly review We’ve seen many of such examples in the data.
  • #33 It might be far-fetched to make any conclusion based on one student’s data, But what if you had thousands of students’ data? === If every student watches it differently, why would combining them all be helpful? Clarify why having the pattern is useful.
  • #34 It might be far-fetched to make any conclusion based on one student’s data, But what if you had thousands of students’ data? === Now that we looked at why data matters, and how data can be used, let me tell you about the specific dataset we used for our analysis.
  • #35 These hot spots in the video may indicate points of confusion or importance. "give me more time" / "I'm not tracking/following" and "this is important" === Re-watching peak: focusing on nonsequential sessions Play peak: points of interest They often correlate, but not always.
  • #37 Qualitative coding ----- Meeting Notes (2/24/14 16:16) ----- one presentation style to another. code example, cut, talking head.
  • #40 Remember that our motivation looking at this data was to find ways to use this data to improve students’ learning experience. Let’s see how the data analysis I presented so far can be connect can be used to achieve this goal. ----- Meeting Notes (10/7/14 17:52) ----- analysis is useful for instructors and video editors, but can we more directly impact students' learning experience? what if we use this data to dynamically change the way video player works?
  • #41 Remember that our motivation looking at this data was to find ways to use this data to improve students’ learning experience. Let’s see how the data analysis I presented so far can be connect can be used to achieve this goal. ----- Meeting Notes (10/7/14 17:52) ----- analysis is useful for instructors and video editors, but can we more directly impact students' learning experience? what if we use this data to dynamically change the way video player works?
  • #43 I’d like to focus on just a few of them.
  • #50 It is an example of control-display ratio adaptation [5, 16, 37], dynamically changing the ratio between physical cursor movement and on-screen cursor movement. The faster the dragging, the weaker the friction. We achieve this effect by temporarily hiding the real cursor, and replacing it with a phantom cursor that moves slower than the real cursor within peak ranges (Figure 3). The idea of enlarging the motor space
  • #69  ----- Meeting Notes (1/28/15 18:46) ----- - simulated navigation tasks? simulated task that people are liekly to encounter. - separate navigation vs learning. - what is within subjects? - say what UI evaluation is normally like: are you good at teaching UI concepts to the audience?
  • #70 Removed “socially-ranked search”
  • #75 Extending this preliminary lab evaluation, I’m currently collaborating with a team at edX to integrate the LectureScape player into the platform. edX instructors will have an option to opt in to use LectureScape as the default video player in our live deployment. Real class Streaming data Cold start problem, adaptive interfaces…
  • #77 “we talk about millions of learners on MOOCs but it’s common for a single video to have millions of viewers.”
  • #78 You can learn how to do almost anything online nowadays by watching how-to videos, including cooking, applying photoshop filters, assembling furniture, and applyingmakeup.
  • #79 How do you go back to a step that you missed for the first time? You have to use the timeline slider to make imprecise estimates. === Q: Why videos in the first place? A: 1) Formative study at Adobe last year showed that it is a preference issue. There are certain types of people who learn better with videos, and they would turn to videos as much as possible, while others use static, step-by-step HTML tutorials. 2) Video captures physical procedures. Step-by-step sometimes skips important steps. limits in navigation affects the learning experience and turns people away from watching video.
  • #80 They have a specific structure, which is that they contain step-by-step instructions. What are some of the properties of how-to videos we can leverage in improving the learning experience?
  • #82 Combining the lessons from the literature, we can conclude that seeing and interacting with the solution structure helps.
  • #84 Ask a question: what are some steps you can think about? May be hard to remember and understand what each step does.
  • #87 ToolScape adds the interactive timeline to let learners click each tool or work-in-progress image to repeat, jump, or skip steps in the workflow.
  • #88 ToolScape adds the interactive timeline to let learners click each tool or work-in-progress image to repeat, jump, or skip steps in the workflow. Wadsworth constant
  • #91  the before rating The self-efficacy measure didn't change much after the task in the baseline condition, whereas the toolscape condition showed a significant increase.
  • #92 In addition to feeling more confident, learners also believed that they performed better with ToolScape.
  • #93 Finally, external raters rated designs produced using ToolScape higher. Lower is better. Considering that it’s the same videos and we didn’t improve the content, The result is quite surprising that the way learners interacted with the material affected the outcome. Summary: "navigational, self-efficacy, and performance benefits of having step-by-step information about the solution"
  • #96 Mention automatic method
  • #99 ToolScape adds the interactive timeline to let learners click each tool or work-in-progress image to repeat, jump, or skip steps in the workflow.
  • #100  ----- Meeting Notes (4/29/14 09:20) ----- So our solution to the annotation challenge is the multi-stage crowdsourcing workflow.
  • #105 Mention DB-Scan
  • #107  ----- Meeting Notes (4/29/14 09:20) ----- We automatically create a multiple choice question
  • #108 How do we pick the label after we break ties?
  • #110 Make it clear why we name it “Expand” Avoid saying “Label”. It’s choosing or identifying images to accompany a step. ----- Meeting Notes (4/29/14 09:20) ----- animate for after?
  • #111 How do we pick the frame after we break ties? Retake the screenshot? ----- Meeting Notes (4/29/14 09:20) ----- why not apply pixel diff and string diff in advance? these are often not identical, so we first wanted to leave it to Turkers, and then apply automatic methods. we might unnecessarily filter things.
  • #114  ----- Meeting Notes (5/1/14 08:11) ----- 1. domain-independent 2. 80% good? obviously it's not perfect. but looking into 20% revealed interesting thing.
  • #118 Crowdy: video learning website with learnersourcing workflow
  • #121 These reflective exercises can be beneficial to learning. Self-explanation, meta-cognition, paying attention.
  • #122 Also useful activity where you get a chance to “compare” multiple alternatives.
  • #123 “Evaluate” the scope of the description.
  • #132 922 subgoals created, 966 upvotes, 527 upvotes
  • #133 922 subgoals created, 966 upvotes, 527 upvotes
  • #134 14 out of 17 learner labels matching or better than expert labels
  • #135 14 out of 17 learner labels matching or better than expert labels
  • #136 14 out of 17 learner labels matching or better than expert labels
  • #137 the choices “...made me feel as though I was on the same wavelength still.”
  • #139 Crowdy: learning activity design this shows a pattern in my learnersourcing research design a meaningful activity, find a way to create something useful
  • #145 Now I’ll briefly mention some of my future research ideas, and wrap up.
  • #146 Many learnersourcing applications that I presented today deal with clickstream data or simple prompts. I think there are exciting opportunities in broadening the scope of learnersourcing to other learning contexts beyond videos. - Programming IDE that learnersources multiple implementations of a same function? - Graphical design tool that learnersources multiple visual assets? - Writing tool that learnersources multiple expressions and phrases? With these ideas, we can make the existing creativity support tools more social and interactive. ===
  • #147 I’ve shown how we can add annotations to existing videos with learnersourcing workflows. Now that we have the enabling technology, what if we had thousands of videos fully annotated and summarized? What interesting applications can we build? Now that videos are indexed at the step level, Look at 100 different ways to perform a step and see what more common and less common approaches are. Based on this data, we can also make search, browsing, and recommendation work at the step level. === Similar for more conceptual lecture videos where students’ alternative explanations can be indexed, browsed, etc.
  • #148 We can also push forward the boundary of learnersourcing by expanding the role of learners. What if the entire course materials are created, taught, and improved by learners? With large-scale explanations, feedback, and improvements that are all learnersourced, Learners take a more active role in learning, and future learners can choose the best set of resources that work for them.
  • #149 Many social domains suffer from the same limitations that video learning does: Members are passive and isolated, there is lack of channel to express individual’s input and contribute in a meaningful way. Learnersourcing presented a model where micro contributions from members of a community can make a difference. This conceptual idea can generalize to various social domains such as open government, nutrition, healthcare, and accessiblity, just to name a few. And the goal is to support more community-driven planning, discussion, decision making, and creative processes. === a lot of community practice and civic issues matter to members, but the decision process and complicated structure are often not accessible to the members. The conceptual idea behind learnersourcing presents a technique for engaging community members in the process, while members engage in natural or voluntary or intrinsically motivated activities.
  • #150 Many social domains suffer from the same limitations that video learning does: Members are passive and isolated, there is lack of channel to express individual’s input and contribute in a meaningful way. Learnersourcing presented a model where micro contributions from members of a community can make a difference. This conceptual idea can generalize to various social domains such as open government, nutrition, healthcare, and accessiblity, just to name a few. And the goal is to support more community-driven planning, discussion, decision making, and creative processes. === a lot of community practice and civic issues matter to members, but the decision process and complicated structure are often not accessible to the members. The conceptual idea behind learnersourcing presents a technique for engaging community members in the process, while members engage in natural or voluntary or intrinsically motivated activities.
  • #152 I started this talk by asking the question of “how can we make video learning really scale?”
  • #153 I think unfortunately this is where we are. The learning experience becomes worse while delivery of content scales up. === As number increases, fully supporting the good ingredients of 1:1 tutoring becomes harder and harder.
  • #154 Many researchers interested in learning at scale are working hard to enable the good components of in-person learning in online settings. And learnersourcing is one such attempt at creating online learning environments that truly scale.
  • #155 But with the unique opportunities we have because of the scale, because of the data, and because of the video, I believe we can be more ambitious and provide an even better learning experience than in-person learning. Learnersourcing is a step toward this vision, by supporting more interactive, collaborative, and data-driven learning. === "beyond being there": technology for distance work, take advantage of new medium. this kind of logic can be applied in my intro.
  • #156 Methods for collecting and processing large-scale data from learners