Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
FOLLOW THE SUCCESSFUL CROWD
RAISING MOOC COMPLETION RATES THROUGH SOCIAL COMPARISON AT SCALE
DAN DAVIS
IOANA JIVET
RENÉ KI...
MOOC LEARNING
FEEDBACK
SOCIAL CUES
SELF-REGULATED LEARNING
METACOGNITION
MOTIVATION
OUTLINE 2. EXPERIMENTS
3. CULTURE
4. CONCLUSION
1. DESIGN
1. DESIGN
FEEDBACK · SOCIAL COMPARISON
SOCIAL COMPARISON THEORY
Establish social and personal worth by comparing
ourselves to others (Festinger, 1954)
Comparison...
ROLE MODELS
learners who earned a passing grade in the
previous edition of the MOOC
FEEDBACK SYSTEM DESIGN
DESIGN GOALS
provide feedback on behavior extracted from edX’s
log traces
afford learners the ability to intuitively under...
Learning tracker
Average graduate You kAverage graduate next week
Time on the
platform
Time watching
videos
Fraction of ti...
FEEDBACK SYSTEM DESIGN
1. Embodied representation of multiple metrics
2. Numerous indicators displayed in a small space
3....
ITERATIVE DESIGN
1. Interactivity
2. Planning ahead
3. Framing
Framing
Feedback Dimension
Planning Ahead
Interactive
2. EXPERIMENTS
RESEARCH QUESTIONS · RESULTS
RQ1
Does social comparison-based learner
feedback lead to increased achievement
and increased self-regulatory behavior
for...
Water Treatment
Urban Sewage
Responsible Innovation
Pre-Calculus
11k
8k
2k
12k
3.5%
3.8%
3.6%
1.7%
42
272
79
327
MOOC ENRO...
5,460
5,483
—
Control
Treatment 1
Treatment 2
4,038
4,099
—
1,184
1,168
—
4,142
4,087
4,065
WaterX SewageX BusinessX CalcX...
WaterX SewageX BusinessX CalcX
1,268
1,251
—
Control
Treatment 1
Treatment 2
771
746
—
164
160
—
1,150
1,147
1,118
ACTIVE:...
RESULTS
COMPLETION · ENGAGEMENT · PRIOR EDU
COURSE
COMPLETION
WaterX
C 1,268 160
MOOC COND. N # PASS PASS RATE
T 1,251 188
12.6%
15.0%
UrbanX
C 771 136
T 746 165
17.6%
22.1%
BusinessX
...
WaterX
C 1,268 160
MOOC COND. N # PASS PASS RATE
T 1,251 188
12.6%
15.0%
UrbanX
C 771 136
T 746 165
17.6%
22.1%
BusinessX
...
WaterX UrbanX BusinessX Overall
Control
Treatment
10%
20%
30%
COURSE
COMPLETION
RQ1
Does social comparison-based learner
feedback lead to increased achievement
and increased self-regulatory behavior for...
RQ1
Does social comparison-based learner
feedback lead to increased achievement
and increased self-regulatory behavior for...
RQ2
Which learners benefit most from the
Feedback System?
PRIOR EDUCATION
WaterX
High 13.2
MOOC
PRIOR
Low 11.8
15.7
11.5
UrbanX
17.4
16.4
21.3
14.0
BusinessX
23.7
21.4
29.1
22.8
OVERALL
16.3
14.4
...
WaterX
High 13.2
MOOC
PRIOR CTRL TREAT
11.8
15.7
11.5
UrbanX
17.4
16.4
21.3
14.0
BusinessX
23.7
21.4
29.1
22.8
OVERALL
16....
RQ2
Which learners benefit most from the
Feedback System?
RQ2
Which learners benefit most from the
Feedback System?
A:
HIGHLY EDUCATED LEARNERS
3. CULTURE*
FRAMING · RESULTS
* PRE-REGISTERED osf.io/ys6au/
RQ3
Does feedback framed in line with a learner’s
cultural context make it more effective?
Framing
Feedback Dimension
Planning Ahead
Interactive
ON TRACK
ON TRACK
BEHIND
Framing
Feedback Dimension
Planning Ahead
Interactive
CULTURE
HOFSTEDE INDIVIDUALISM
INDIVIDUALISM
INDIVIDUAL AS AN INDEPENDENT
ACTOR WITH LOOSE SOCIAL RELATIONS
COLLECTIVISM
CLOSELY KNIT SOCIAL RELATIONS AND
GREAT CARE FOR THE WELL-BEING OF ALL
INDIVIDUALISMCOLLECTIVISM
CULTURE
HOFSTEDE INDIVIDUALISM
GELFAND TIGHTNESS
HAVE MANY STRONGLY ENFORCED RULES
AND LITTLE TOLERANCE FOR DEVIANCE
TIGHTLOOSE
HAVE FEW STRONGLY ENFORCED RULES
AND GREATER TOLERANCE FOR DEVIANCE
TIGHTLOOSE
RQ3
Does feedback framed in line with a
learner’s cultural context make it more
effective?
RQ3
Does feedback framed in line with a
learner’s cultural context make it more
effective?
Feedback in line with a learner...
RQ3
Does feedback framed in line with a
learner’s cultural context make it more
effective?
Feedback in line with a learner...
WaterX
MOOC PASS RATE
13.8%
UrbanX 19.8%
BusinessX 31%
CalcX 5%
RAISE ACTIVITY THRESHOLD
5 MINUTES -> 1 HOUR
From N = 3,415 to N = 658
WaterX
MOOC PASS RATE
13.8%
UrbanX 19.8%
BusinessX 31%
CalcX 16%
INDIVIDUALIST FRAMING
+ Activity Threshold
Increased completion rates across cultural contexts
Control: 12.8%
Individualis...
INDIVIDUALIST FRAMING
+ Activity Threshold
Increased completion rates across cultural contexts
Control: 12.8%
Individualis...
5. CONCLUSION
WHAT’S NEXT
1. REAL-TIME
2. FRAMING
3. PERSONAS
1. REAL-TIME
2. FRAMING
3. PERSONAS
1. REAL-TIME
2. FRAMING
3. PERSONAS
THANK YOU
BIT.LY/WIS-LEARNING-ANALYTICS
LAMBDA-LAB
DAN DAVIS
IOANA JIVET
RENÉ KIZILCEC
GUANLIANG CHEN
CLAUDIA HAUFF
GEERT-...
LOW CULTURAL TIGHTNESS
PREDICTS COMPLETION
p < 0.02
Control 1,150 45
CONDITION N # PASS PASS RATE
3.9%
Indiv. - Promotion 1,147 62 5.4%
Collect. - Prevent. 1,118 51 4.5%
OVER...
Upcoming SlideShare
Loading in …5
×

Follow the Successful Crowd: Raising MOOC Completion Rates through Social Comparison at Scale

474 views

Published on

Presented at the 7th International Conference of Learning Analytics and Knowledge (LAK '17) in Vancouver, Canada. Friday March 17, 2017.

Published in: Education
  • Be the first to comment

Follow the Successful Crowd: Raising MOOC Completion Rates through Social Comparison at Scale

  1. 1. FOLLOW THE SUCCESSFUL CROWD RAISING MOOC COMPLETION RATES THROUGH SOCIAL COMPARISON AT SCALE DAN DAVIS IOANA JIVET RENÉ KIZILCEC LAMBDA-LAB GUANLIANG CHEN CLAUDIA HAUFF GEERT-JAN HOUBEN
  2. 2. MOOC LEARNING
  3. 3. FEEDBACK
  4. 4. SOCIAL CUES SELF-REGULATED LEARNING METACOGNITION MOTIVATION
  5. 5. OUTLINE 2. EXPERIMENTS 3. CULTURE 4. CONCLUSION 1. DESIGN
  6. 6. 1. DESIGN FEEDBACK · SOCIAL COMPARISON
  7. 7. SOCIAL COMPARISON THEORY Establish social and personal worth by comparing ourselves to others (Festinger, 1954) Comparison with slightly-better-performing peers has a beneficial effect on students’ grades (Huguet, 2001)
  8. 8. ROLE MODELS learners who earned a passing grade in the previous edition of the MOOC
  9. 9. FEEDBACK SYSTEM DESIGN
  10. 10. DESIGN GOALS provide feedback on behavior extracted from edX’s log traces afford learners the ability to intuitively understand and explore the information presented (Kay) show learners specific ways to take action and change their behavior Traceable Scrutable Actionable
  11. 11. Learning tracker Average graduate You kAverage graduate next week Time on the platform Time watching videos Fraction of time spent watching videos while on the platform Videos watched Quiz answers submitted Timeliness of quiz answer submission
  12. 12. FEEDBACK SYSTEM DESIGN 1. Embodied representation of multiple metrics 2. Numerous indicators displayed in a small space 3. Easily comparable
  13. 13. ITERATIVE DESIGN 1. Interactivity 2. Planning ahead 3. Framing Framing Feedback Dimension Planning Ahead Interactive
  14. 14. 2. EXPERIMENTS RESEARCH QUESTIONS · RESULTS
  15. 15. RQ1 Does social comparison-based learner feedback lead to increased achievement and increased self-regulatory behavior for MOOC learners?
  16. 16. Water Treatment Urban Sewage Responsible Innovation Pre-Calculus 11k 8k 2k 12k 3.5% 3.8% 3.6% 1.7% 42 272 79 327 MOOC ENROLLED PASS QUIZ QUESTIONSRATE
  17. 17. 5,460 5,483 — Control Treatment 1 Treatment 2 4,038 4,099 — 1,184 1,168 — 4,142 4,087 4,065 WaterX SewageX BusinessX CalcX ALL ENROLLED
  18. 18. WaterX SewageX BusinessX CalcX 1,268 1,251 — Control Treatment 1 Treatment 2 771 746 — 164 160 — 1,150 1,147 1,118 ACTIVE: >5 MIN IN COURSE
  19. 19. RESULTS COMPLETION · ENGAGEMENT · PRIOR EDU
  20. 20. COURSE COMPLETION
  21. 21. WaterX C 1,268 160 MOOC COND. N # PASS PASS RATE T 1,251 188 12.6% 15.0% UrbanX C 771 136 T 746 165 17.6% 22.1% BusinessX C 164 46 T 160 54 28.0% 33.8% OVERALL C 2,203 342 T 2,157 407 15.5% 18.9%
  22. 22. WaterX C 1,268 160 MOOC COND. N # PASS PASS RATE T 1,251 188 12.6% 15.0% UrbanX C 771 136 T 746 165 17.6% 22.1% BusinessX C 164 46 T 160 54 28.0% 33.8% OVERALL C 2,203 342 T 2,157 407 15.5% 18.9% **
  23. 23. WaterX UrbanX BusinessX Overall Control Treatment 10% 20% 30% COURSE COMPLETION
  24. 24. RQ1 Does social comparison-based learner feedback lead to increased achievement and increased self-regulatory behavior for MOOC learners? Providing learners a comparison of their own behavior to that of previously successful peers will increase learner achievement. H1.1
  25. 25. RQ1 Does social comparison-based learner feedback lead to increased achievement and increased self-regulatory behavior for MOOC learners? Providing learners a comparison of their own behavior to that of previously successful peers will increase learner achievement and engagement H1.1 Learners will change the aspects of their behavior that the Feedback System makes them aware of. H1.2 X
  26. 26. RQ2 Which learners benefit most from the Feedback System?
  27. 27. PRIOR EDUCATION
  28. 28. WaterX High 13.2 MOOC PRIOR Low 11.8 15.7 11.5 UrbanX 17.4 16.4 21.3 14.0 BusinessX 23.7 21.4 29.1 22.8 OVERALL 16.3 14.4 19.5 13.6 High Low High Low High Low EDU 0.15 0.16 0.04 0.66 0.04 0.78 <0.01 0.36 CTRL TREAT p GRADE GRADE
  29. 29. WaterX High 13.2 MOOC PRIOR CTRL TREAT 11.8 15.7 11.5 UrbanX 17.4 16.4 21.3 14.0 BusinessX 23.7 21.4 29.1 22.8 OVERALL 16.3 14.4 19.5 13.6 High High High EDU p 0.15 0.16 0.04 0.66 0.04 0.78 <0.01 0.36 < < < < GRADE GRADE Low Low Low Low
  30. 30. RQ2 Which learners benefit most from the Feedback System?
  31. 31. RQ2 Which learners benefit most from the Feedback System? A: HIGHLY EDUCATED LEARNERS
  32. 32. 3. CULTURE* FRAMING · RESULTS * PRE-REGISTERED osf.io/ys6au/
  33. 33. RQ3 Does feedback framed in line with a learner’s cultural context make it more effective?
  34. 34. Framing Feedback Dimension Planning Ahead Interactive
  35. 35. ON TRACK
  36. 36. ON TRACK
  37. 37. BEHIND
  38. 38. Framing Feedback Dimension Planning Ahead Interactive
  39. 39. CULTURE HOFSTEDE INDIVIDUALISM
  40. 40. INDIVIDUALISM INDIVIDUAL AS AN INDEPENDENT ACTOR WITH LOOSE SOCIAL RELATIONS COLLECTIVISM
  41. 41. CLOSELY KNIT SOCIAL RELATIONS AND GREAT CARE FOR THE WELL-BEING OF ALL INDIVIDUALISMCOLLECTIVISM
  42. 42. CULTURE HOFSTEDE INDIVIDUALISM GELFAND TIGHTNESS
  43. 43. HAVE MANY STRONGLY ENFORCED RULES AND LITTLE TOLERANCE FOR DEVIANCE TIGHTLOOSE
  44. 44. HAVE FEW STRONGLY ENFORCED RULES AND GREATER TOLERANCE FOR DEVIANCE TIGHTLOOSE
  45. 45. RQ3 Does feedback framed in line with a learner’s cultural context make it more effective?
  46. 46. RQ3 Does feedback framed in line with a learner’s cultural context make it more effective? Feedback in line with a learner’s cultural individualism index will increase its impact on the learner H3.1 X
  47. 47. RQ3 Does feedback framed in line with a learner’s cultural context make it more effective? Feedback in line with a learner’s cultural individualism index will increase its impact on the learner H3.1 Feedback in alignment with a learner’s cultural tightness will increase its impact on the learner H3.2 X
  48. 48. WaterX MOOC PASS RATE 13.8% UrbanX 19.8% BusinessX 31% CalcX 5%
  49. 49. RAISE ACTIVITY THRESHOLD 5 MINUTES -> 1 HOUR From N = 3,415 to N = 658
  50. 50. WaterX MOOC PASS RATE 13.8% UrbanX 19.8% BusinessX 31% CalcX 16%
  51. 51. INDIVIDUALIST FRAMING + Activity Threshold Increased completion rates across cultural contexts Control: 12.8% Individualist Framing: 19.9% 55% Increase (t = 2.02, p = 0.04, N = 441)
  52. 52. INDIVIDUALIST FRAMING + Activity Threshold Increased completion rates across cultural contexts Control: 12.8% Individualist Framing: 19.9% 55% Increase (t = 2.02, p = 0.04, N = 441) Increased completion rates especially in tight cultures Control: 12.1% Individualist Framing: 36.3% 300% Increase (t = 2.07, p = 0.04, N = 55)
  53. 53. 5. CONCLUSION WHAT’S NEXT
  54. 54. 1. REAL-TIME 2. FRAMING 3. PERSONAS
  55. 55. 1. REAL-TIME 2. FRAMING 3. PERSONAS
  56. 56. 1. REAL-TIME 2. FRAMING 3. PERSONAS
  57. 57. THANK YOU BIT.LY/WIS-LEARNING-ANALYTICS LAMBDA-LAB DAN DAVIS IOANA JIVET RENÉ KIZILCEC GUANLIANG CHEN CLAUDIA HAUFF GEERT-JAN HOUBEN
  58. 58. LOW CULTURAL TIGHTNESS PREDICTS COMPLETION p < 0.02
  59. 59. Control 1,150 45 CONDITION N # PASS PASS RATE 3.9% Indiv. - Promotion 1,147 62 5.4% Collect. - Prevent. 1,118 51 4.5% OVERALL EFFECT

×