Downloaded 25 times
Uploaded byIoana Jivet
The Learning Tracker - A Learner Dashboard that Encourages Self-regulation in MOOC Learners
The document presents a study on the 'Learning Tracker', a dashboard designed to enhance self-regulation among MOOC learners by providing feedback through social comparisons with successful peers. The study, which involved randomized controlled trials across three MOOCs, demonstrated improved graduation rates and engagement with graded materials; however, it found no immediate effects on learners' self-regulation behaviors. Future work is suggested for testing personalized feedback and exploring social effects on motivation and learning behavior.
More Related Content
Similar to The Learning Tracker - A Learner Dashboard that Encourages Self-regulation in MOOC Learners
The Learning Tracker - A Learner Dashboard that Encourages Self-regulation in MOOC Learners
- 1. 1 The Learning Tracker ALearner Dashboard that Encourages Self-Regulation in MOOC Learners Ioana Jivet September 19th, 2016
- 2. 2 Agenda • Motivation • LearningTracker design • Experimental setup • Results • Conclusion and outlook
- 3.
- 4. 4 What is aMOOC? Massive Open Online Course Best Courses. Top Institutions. Learn anytime, anywhere. • 35 million learners • 500 universities • 4 200 MOOCs
- 5. 5 Dropout as amain challenge • Low completion rates <15 % (Jordan, 2016) • Underdeveloped learning skills and study habits – High autonomy – Role of the teacher – Low metacognitive awareness
- 6. 6 Self Regulated Learning •Definition: capability of the learner “to adjust her actions and goals to achieve desired results in light of changing environmental conditions” (Zimmerman, 1990) • Major success factor in online learning environments, including MOOCs • Lack of learner support in current MOOC platforms
- 7. 7 Learner support onMOOC platforms - edX
- 8. 8 Aim Investigate how self-regulated learningskills can be enhanced in MOOC learners Encouraging metacognition and self-reflection on learning behaviour Providing feedback through social comparison with successful learners on a learner dashboard
- 9.
- 10. 10 Development Design-based research methodology •Incremental • Evaluation on edX MOOCs offered by TU Delft Two components • Data • Visualisation First iteration Evaluation January – March 2016 Second iteration Evaluation April – June 2016
- 11. 11 Behaviour metrics Data edX trace logs Widget Examples ofmetrics displayed on the widget • Number of graded quizzes attempted • Number of forum visits • Timeliness of assignment submission Social comparison with the average graduate
- 12.
- 13. 13 Preliminary evaluation ofthe first iteration • Metric configuration • Additional information set – Average graduate in the following week – Reflection and planning support Adjustments in the second iteration
- 14.
- 15. 15 Preliminary evaluation ofthe first iteration • Metric configuration • Additional information set – Average graduate at the end of current week – Reflection and planning support • Interactive elements Adjustments in the second iteration
- 16. 16 Interactive elements –information sets
- 17. 17 Interactive elements -tooltip
- 18. 18 Implementation of thewidget edX trace logs Behaviour metrics Widget
- 19.
- 20. 20 Experimental setup Three TUDelft MOOCs – Weekly publication of learning material – Video lectures, weekly assignments, practice quizzes – Graduation: >60% final score Replicated longitudinal study
- 21. 21 Experimental setup Method: randomizedcontrolled trial – Demographic analysis to ensure populations are sufficiently randomized WaterX SewageX InnovationX Test group 5 460 4 038 1 184 Control group 5 483 4 099 1 168 Total enrolled 10 943 8 137 2 352
- 22.
- 23. 23 Learners’ performance RQ1 Arelearners more likely to complete the course when they can compare their behaviour to that of previous graduates?
- 24. 24 Learners’ performance –graduation Higher graduation rate.
- 25. 25 Learners’ performance –final grades More learners graduate, but they do not pursue higher grades.
- 26. 26 Learners’ performance –final grades More learners graduate, but they do not pursue higher grades.
- 27. 27 Learners’ behaviour RQ2.1 Dolearners become more engaged with the MOOC when they can compare their behaviour with that of successful learners?
- 28. 28 Learners’ engagement –course material Learners are more engaged with the graded course material. WaterX SewageX Innovationx Graded quizzes .036 .114 .044 Practice non-graded quizzes .512 .071 - Mann-Whitney test results (p-values) between the test group and the control group. – Significance level α = .050 – Significant differences are marked in bold.
- 29. 29 More learners areengaged with graded course content. Learners’ engagement – course material
- 30. 30 Learners’ engagement –course material More learners are engaged with graded course content.
- 31. 31 Learners’ self-regulation RQ2.2 Dolearners show improvement of their time-management skills when they compare their behaviour to that of successful learners?
- 32. 32 Learners’ self-regulation -procrastination WaterX SewageX Innovationx Timeliness (recommended) .055 .113 .039 Timeliness (actual) .040 .145 .035 Mann-Whitney test results (p-values) between the test group and the control group. – Significance level α = .050 – Significant differences are marked in bold. Learners procrastinate less.
- 33. 33 Learners’ self-regulation –procrastination Learners procrastinate less.
- 34. 34 Learners’ on-trackness RQ2.3 Dolearners change their behaviour so it becomes similar to that of successful learners when they compare themselves to it?
- 35. 35 Learners’ on-trackness Similarity betweena learners’ behaviour and that of the average graduate 1. Compute on-trackness score weekly 2. Cluster learners based on the evolution of the on-trackness score
- 36. 36 Learners’ on-trackeness –clusters No conclusive evidence that the Learning Tracker influences the distribution of learners into clusters. on-track behind, but keep up behind, initial activity behind, no activity
- 37.
- 38. 38 Overall outcomes • Higherlikelihood of graduation • No immediate effect on self-regulated behaviour (e.g. procrastination) • Limited feedback affects behaviour • On-trackness classification based on behaviour similarity
- 39. 39 Future work Testing differentdefinitions for success Personalized feedback (demographics) Social effects – behaviour uniformization – motivation Extensive longitudinal evaluations
- 40. 40 Demonstration paper Davis, Chen,Jivet, Hauff, & Houben, 2016: Encouraging Metacognition & Self-Regulation in MOOCs through Increased Learner Feedback – In Learning Analytics and Knowledge 2016 Learning Analytics for Learners Workshop.
- 41.
- 42.
- 43. 43 Learners’ on-trackness -score Arithmetic weighted sum of metric deviations
- 44. 44 Learners’ on-trackness -score 1. Behind
- 45. 45 Learners’ on-trackness -score 1. Behind 2. On-track
- 46. 46 Learners’ on-trackness -score 1. Behind 2. On-track 3. Ahead
- 47. 47 Inspiration – SearchDashboard Impact of reflection and social comparison on search behavior (Bateman, 2012) Reference model – expert searchers
- 48. 48 Behaviour metrics usedon the widget
- 49.
Editor's Notes
- #5 MOOC stands for Massive Open Online Course. The term massive refers to the large number of learners that can participate. MOOCs are open, meaning that learners have free access to the content once they enroll. The term online refers to the fact that accessing a MOOC can only be done by somebody that has Internet access. MOOCs caught public eye in 2012, when top universities like Stanford, MIT or Harvard launched today’s largest MOOC platforms. One such example is edX, a nonprofit MOOC provider founded by Harvard University and MIT. Their mission is to increase access to high-quality education for anyone, anywhere. MOOCs are expected to revolutionize education by making high quality education accessible to the masses and thus reducing the gap between the most privileged and the most disadvantaged learners. Until now, more than 35 million learners enrolled in at least one MOOC and more than 500 universities are offering over 4000 MOOCs.
- #6 Despite their potential, MOOCs face some challenges. One of these major challenges is a low completion rate most of the time below 15%. Although literature identifies several reasons for learners dropping out early, the one that we want to focus on is underdeveloped learning skills and study habits. Very often, learners drop out because they are not equipped with proper skills to learn with a MOOC. A new learning environment that requires high autonomy in terms of motivation, defining learning paths and engaging with other MOOC participants. Learners have a lot of freedom in choosing when, where, what and how to learn. Learners face less constraints than in traditional face-to-face education. There are no consequences for failing or dropping out and the role of the instructor/teachers changes. Many times learners are not aware that their learning skills are not adequate. Low metacognitive awareness means that learners are not inclined to think about and evaluate their own thinking process or the effectiveness of their strategies.
- #7 Yet, there are solutions. Learning psychologists identified that what makes learners successful is a skill called self-regulated learning. They explain that this is a major factor that influences the large-scale success/failure of online learning environments, including MOOCs However, current MOOC platforms fail to support learners in developing these skills which are indispensable for online learning.
- #8 Looking at edX, for example, learners have the Progress page on which theys can view the grades they obtained for each assignment, but they do not receive any information of how they could improve their learning.
- #9 Thus the aim of this work is to address the lack of support in the learning process offered to learners and investigate how SRL skills can be enhanced in MOOC learners. Relying on research done in the field of learning sciences, we hypothesize that SRL skills can be developed if learners reflect on and evaluate their learning behaviour. In order to test this hypothesis, we developed a learner dashboard embedded on edX MOOC pages on which learners receive feedback on their learning behaviour. At the same time, learners can compare their behaviour to that of previously successful learners. What we considered “successful learners” are learners that graduated previous editions of the same MOOC.
- #11 Incremental development: First iteration in January-March 2016 and a second iteration in April-June 2016
- #12 How it works: based on the edX trace logs that record every action learners perform on the platform, we extract and compute a series of metrics that are displayed on a widget embedded in the course pages.
- #13 Reasons for choosing a spider chart: concise visualization of numerous metrics in a small space and easily comparable medium 6 metrics are displayed around the spider chart. Learners can visualise their own performance and the one of the average graduate. The values of the metrics are computed once, before the beginning of the course for the average graduate. The values for the current learners are updated once a week when new edX data becomes available.
- #14 The first iteration was then evaluated on a live TU Delft MOOC on edX. Based on preliminary results that showed that the LT had a positive effect on the graduation rates, several adjustments were made to the second iteration. First off, the metrics were changed to focus more on self-regulating behaviour. For example, we included metrics that referred to the number of visits per week, the average time between two consecutive sessions or the average length of a session.
- #19 Regarding the technologies used to implement it, for computing the behaviour metrics, we used code written in Java 8. The widget in itself is a JS Script and for drawing it we used Highcharts, a JS charting library.
- #21 What gives validity to our study is that we evaluated two iteration of the LT over the full duration of each MOOC. Such studies are rare in literature.
- #22 Method: Learners are randomly assigned to a test or a control group. Test group has access to the LT, while the control group does not. This method allowed us to identify the effects of the Learning Tracker by comparing the behaviour of the test group with that of the control group.
- #23 The analysis was performed only taking into account data from active learners (those that spent more than 5 minutes on the platform).
- #25 Here we plotted the percentage of learners that graduated in each group in all three MOOCs. As we can see, the graduation percentage is higher in all three cases. No significant differences between the final grades.
- #26 We also looked at the final grades the learners obtained. Statistical tests show no significant differences between the test group and the control group. This plot show the distribution of learners in the test group and the control group according to the final grade. Two things to notice: Firstly, there is a high density around 0 – common to every MOOC Secondly, the curve representing the test group is above the one representing the control group – meaning that more learners passed the graduation threshold, although they did not pursue higher grades.
- #27 The same results hold also for the other two course.
- #28 The next steps of the analysis looked into what could be reasons for this change in graduation rate. Did the learners behaviour change? And if so, in what ways?
- #29 Solving practice quizzes first to grasp concepts is not a strategy employed by the test learners. - Might be an explanation for the higher % of learners in the range of 60-80% for the final grade.
- #30 This graph shows the progression of both group through the first MOOC with respect to the number of learners that attempted at least one graded quiz question since the beginning of the course. The widget was made available in week 2.
- #31 Number of learners that attempted at least one graded quiz question since the beginning of the course. The difference becomes visible between week 2 and 3, a week after the widget was made available.
- #32 We investigated how learners use their time on the platform and their time-management skills
- #33 We observed significant differences in one metric: the timeliness of submission = the average time between learners submit assignments and the deadline.
- #36 To evaluate learners’ on-trackness, we measured the similarity between a learners’ behaviour and that of a successful learners by calculating an on-trackness score.
- #37 Using k-means clustering algorithm, we grouped learners into clusters with similar behaviours over time. The learners exhibit similar behaviour patterns in all three courses. - Describe the clusters Out of the four patterns, two (Cluster 1 and Cluster 2) show a steady progress, while the other two exhibit a decrease in on-trackness over time (Cluster 3 and Cluster 4). The decrease in on-trackness score reflects drop-outs or very low activity. However, when we looked at the distribution of the test and control learners into the four clusters we did not find any evidence that shows that the LT motivates learners to be on-track.
- #39 Learning behaviour can be broken down into several study habits that influence each other Learning behaviour is a “web” of study habits
- #40 Positive effect: (a) Increases the likelihood of graduation because it increases the engagement with graded course material. (b) Reduces procrastination.
- #44 on-trackness score = the similarity between one’s behaviour and that of successful learners To evaluate learners’ on-trackness, we measured the similarity between a learners’ behaviour and that of a successful learners by calculating an on-trackness score. The score is computed through an arithmetic weighted sum of the metric deviations. The metric deviations are the differences between a learner’s scaled value of the metric and the average graduate’s scaled value. The weights are inversely proportional with the amplitude of the metric deviation. The higher the difference, the lower the weight.
- #45 on-trackness score = the similarity between one’s behaviour and that of successful learners To evaluate learners’ on-trackness, we measured the similarity between a learners’ behaviour and that of a successful learners by calculating an on-trackness score. The score is computed through an arithmetic weighted sum of the metric deviations. The metric deviations are the differences between a learner’s scaled value of the metric and the average graduate’s scaled value. The weights are inversely proportional with the amplitude of the metric deviation. The higher the difference, the lower the weight.
- #46 on-trackness score = the similarity between one’s behaviour and that of successful learners To evaluate learners’ on-trackness, we measured the similarity between a learners’ behaviour and that of a successful learners by calculating an on-trackness score. The score is computed through an arithmetic weighted sum of the metric deviations. The metric deviations are the differences between a learner’s scaled value of the metric and the average graduate’s scaled value. The weights are inversely proportional with the amplitude of the metric deviation. The higher the difference, the lower the weight.
- #47 on-trackness score = the similarity between one’s behaviour and that of successful learners To evaluate learners’ on-trackness, we measured the similarity between a learners’ behaviour and that of a successful learners by calculating an on-trackness score. The score is computed through an arithmetic weighted sum of the metric deviations. The metric deviations are the differences between a learner’s scaled value of the metric and the average graduate’s scaled value. The weights are inversely proportional with the amplitude of the metric deviation. The higher the difference, the lower the weight.
- #48 The inspiration for this work was research done in the field of search engines. Bateman investigated the effects of reflection and social comparison in search behaviour with positive results. As a reference model, they quantified the behaviour of “expert searchers”. In our case, what we consider a model worth comparing against are previous graduates of the same MOOC.