The researchers at TU Delft are studying MOOC learners and their behaviors through various research projects aimed at gaining insights into learning at scale. Their goals are to better understand learners, identify patterns in their behaviors, and design interventions to support adaptive learning. Some of their key findings include identifying subsets of learners who display learning transfer across courses, linking learners' online behaviors to personality traits, and analyzing how well learners adhere to the prescribed learning paths in MOOCs. The researchers are exploring ways to provide personalized feedback and encourage self-regulated learning at scale, with the aim of helping learners improve their performance.

1. MOOC RESEARCH AT TU DELFT
THE LEARNING ENVIRONMENT & BEYOND
DAN DAVIS
TU DELFT
WEB INFORMATION SYSTEMS GROUP, LAMBDA LAB LAMBDA-LAB

2. WHO ARE WE?
Guanliang Chen
PhD Researcher
Dan Davis
PhD Researcher
Claudia Hauff
Assistant Prof.
Geert-Jan Houben
Professor
Research backgrounds in user modeling, information retrieval, big data processing
and learning technologies.

3. OUR GOALS
1. Gain actionable insights into learner behaviors at scale
2. Increase our knowledge about learners by looking
beyond the learning platform
3. Design and implement interventions that enable
adaptive learning at scale

4. OUTLINE1. Beyond the learning platform · WebSci ’16
2. Learning transfer · L@S ’16
3. Personality in MOOCS · UMAP ’16
4. Learners -> Earners · In Progress
5. Learning Paths · EDM ’16
6. Study Strategies · EC-TEL ’16
7. Feedback- Pilot · LAK ‘16 Workshop
8. Adaptive Feedback · In Progress
LearnerModelingSRL&Support

5. 1. BEYOND THE LEARNING PLATFORM
· Can we link our learners to their social web accounts?
· Do social Web platforms enable us to observe user attributes
relevant to the online learning experience?
Guanliang Chen, Dan Davis, Jun Lin, Claudia Hauff, and Geert-Jan Houben. Beyond the MOOC platform:
Gaining Insights about Learners from the Social Web. ACM WebScience, pp. 15-24, 2016.
Over 1M
learners
2—42%
ID’d / course

6. 2. LEARNING TRANSFER:
Major findings:
· A small subset of learners (8%) display learning transfer.
· Existing learning transfer findings from the workplace
& classroom setting mostly also hold in MOOCs.
Guanliang Chen, Dan Davis, Claudia Hauff, and Geert-Jan Houben, Learning Transfer: does it take place in MOOCs?,
ACM Learning At Scale, pp. 409-418, 2016.
do learners apply what they learned in practice?

7. 3. PERSONALITY IN ONLINE LEARNING
Guanliang Chen, Dan Davis, Claudia Hauff, and Geert-Jan Houben. On the Impact of Personality in Massive Open Online
Learning. ACM UMAP 2016.
Also to appear at LWMOOCs III 2016.
· Does personality impact learner engagement, learner
behavior and learner success in the context of MOOCs?
· Can learners’ personalities be predicted based on their
behaviors exhibited in a MOOC platform?

8. 4. FROM LEARNERS TO EARNERS
· Are MOOC learners able to solve real-world paid tasks
from an online work platform with sufficient quality?
(In Press) Guanliang Chen, Dan Davis, Markus Krause, Efthimia Aivaloglou, Claudia Hauff, and Geert-Jan
Houben. From Learners to Earners: Enabling MOOC Learners to Apply their Skills and Earn Money in an
Online Market Place.
· Currently building recommender system for scalable
implementation in TU DelftX Data Analysis MOOC

9. 5. EXECUTED VS. DESIGNED LEARNING PATH
Dan Davis, Guanliang Chen, Claudia Hauff, and Geert-Jan Houben. Gauging MOOC Learners’ Adherence to the Designed
Learning Path. EDM 2016.
To what extent do learners adhere to a MOOC’s designed
learning path?
1. VIDEO INTERACTIONS
2. BEHAVIOR PATTERN CHAINS
3. EVENT TYPE TRANSITIONS

10. 1. VIDEO INTERACTIONS
DO LEARNERS WATCH VIDEO LECTURES IN THE PRESCRIBED ORDER?

11. VIDEO INTERACTIONS
Non-Passing
Passing
Week 1 Week 2
Week 3
FP101x
Designed Lecture Order
Executed Paths

12. 2. BEHAVIOR PATTERN CHAINS
WHAT ARE THE MOST COMMON SEQUENCES IN THE EXECUTED LEARNING PATHS?

13. VIDEO QUIZ PROGRESS FORUM
WATCH START VIEW START
SUBMIT
END
SUBMIT
END

14. EIGHT-STEP CHAIN
FORUMEND
EXAMPLE
LECTUREWATCH QUIZSTART QUIZSUBMIT QUIZSUBMIT
QUIZEND PROGRESS LECTUREWATCH

15. MOTIF
A RECURRING OR REPEATED ELEMENT

16. MOTIF FREQ
TOTAL
FREQ
PASS
FREQ
FAIL
QUIZ COMPLETE 552,363
(29.4%)
328,995
(30.8%)
223,368
(27.7%)
1.
BINGE WATCHING 149,784
(8%)
59,498
(5.6%)
90,286
(11.2%)
2.
LECTURE -> QUIZ COMPLETE 100,179
(5.3%)
50,415
(4.7%)
49,764
(6.2%)
3.
QUIZ COMPLETE -> FORUM 99,828
(5.3%)
67,722
(6.3%)
32,106
(4%)
4.
FP101x
XQUIZ events only with at least 1 X = SUBMIT
WATCH events only
WATCH event(s) followed by XQUIZ events w/ >1 X = SUBMIT
XQUIZ events w/ >1 X = SUBMIT followed by XFORUM events

17. 3. EVENT TYPE TRANSITIONS
HOW DO STUDENTS NAVIGATE BETWEEN CERTAIN EVENT TYPES?

18. EVENT TYPE TRANSITIONS
Frame101x Non-Passing
Markov Model
State Visualization
(EXECUTED)

19. EVENT TYPE TRANSITIONS
Frame101x Non-Passing
(EXECUTED)
Frame101x Passing

20. EVENT TYPE TRANSITIONS(EXECUTED)
Frame101x Non-Passing Frame101x Passing32%
35%
VIDEO
QUIZ
END
QUIZ
SUBMIT
FORUM
SUBMIT
FORUM
END
PROGRESS
FORUM
START
QUIZ
START VIDEO
QUIZ
END QUIZ
SUBMIT
FORUM
SUBMIT
FORUM
END
PROGRESS
FORUM
START
QUIZ
START
21%
28%
44%
44%

21. 6. RETRIEVAL PRACTICE & STUDY PLANNING IN
MOOCS
D. Davis, G. Chen, Tim v.d. Zee, C. Hauff, and G.J. Houben. Retrieval Practice and Study Planning in MOOCs: Exploring
Classroom-Based Self-Regulated Learning Strategies at Scale. ECTEL 2016.
· To what extent can findings from the learning sciences apply to
MOOC learners?
· Do learners engage with self-regulated learning (SRL)
interventions?
· Does the engagement have an effect on learners’ test
performance?

22. 6. RETRIEVAL PRACTICE & STUDY PLANNING IN
MOOCS
D. Davis, G. Chen, Tim v.d. Zee, C. Hauff, and G.J. Houben. Retrieval Practice and Study Planning in MOOCs: Exploring
Classroom-Based Self-Regulated Learning Strategies at Scale. ECTEL 2016.
Retrieval Practice Study Planning

23. 6. RETRIEVAL PRACTICE & STUDY PLANNING IN
MOOCS
D. Davis, G. Chen, Tim v.d. Zee, C. Hauff, and G.J. Houben. Retrieval Practice and Study Planning in MOOCs: Exploring
Classroom-Based Self-Regulated Learning Strategies at Scale. ECTEL 2016.
Retrieval Practice Study Planning
RESULTS: NONCOMPLIANCE
(NEED MORE ENGAGING INTERVENTIONS)

24. 7. PERSONALIZED FEEDBACK @ SCALE
Dan Davis, Guanliang Chen, Ioana Jivet, Claudia Hauff, and Geert-Jan Houben. Encouraging Metacognition & Self-
Regulation in MOOCs through Increased Learner Feedback. LAL Workshop 2016.
ENCOURAGING METACOGNITION & SRL THROUGH INCREASED LEARNER FEEDBACK
Do learners change their behavior when confronted with their
learning performance relative to that of successful learners?

25. 8. PERSONALIZED FEEDBACK @ SCALE
(In Progress)
ENCOURAGING METACOGNITION & SRL THROUGH INCREASED LEARNER FEEDBACK
Do learners change their behavior when confronted with their
learning performance relative to that of successful learners?

26. LOOKING FORWARD
· NEW INNOVATIONS IN FEEDBACK
-> What & how do learners want & need
-> Value in cultural context?
· PATTERN MINING
-> Positive deviance
-> New sub-groupings (beyond passing v. non-passing)
-> How can we nudge to a “better” path (& should we?)
· BEYOND THE PLATFORM
-> Where can we look to gain meaningful learner insights?
-> What can we do with this information?

27. THANK YOU
BIT.LY/WIS-LEARNING-ANALYTICS
LAMBDA-LAB
DAN DAVIS
TU DELFT
WEB INFORMATION SYSTEMS GROUP, LAMBDA LAB