This document describes the development of a data-driven learning interest recommendation system on MOOCs. It discusses using students' activity logs on MOOCs, such as video clickstreams, to analyze their interests and provide recommendations to help students. The system, called Videomark, analyzes video activity data to identify keywords and popular video segments for each week's content. It displays these as a keyword cloud and allows students to click keywords to access related video sections. An experiment on a networking security course found Videomark improved students' learning performance, engagement and experience compared to not having the recommendation system. Future work could include generating keywords in other languages and improving the user interface.

1. 1 / 31
Developing a Data-Driven Learning Interest Recommendation System on
MOOCs
Hsuan-Ming Chang
Nen-Fu Huang
National Tsing Hua University
Institute of Computer Science
High Speed Network Lab

2. 2 / 31
Outline
❖ Introduction
❖ Implementation
❖ Experimental Results
❖ Conclusion
❖ Future Work

3. 3 / 31
Motivation
❖ Students on MOOCs are much more than teaching
group.
❖ Traditionally, students on MOOCs may encounter
some problems.
▪ At the beginning of the course, instructional videos
have too many points to focus on.
▪ After watching instructional videos, it often lacks in
systematic way to review course contents.
▪ If student wants to focus on specific concepts, current
system can’t help student to collect all relative contents
for boosting learning efficiency.

4. 4 / 31
Motivation
❖ Massive student activity logs
▪ Students’ activity records can be saved and analyzed.
▪ We can make use of students’ video click stream
records.
▪ After analyze activity records on MOOCs, feedback to
students to help them.

5. 5 / 31
❖ How can these activity logs be collected and
analyzed ?
❖ How can we use these analyzed result to help
students ?
How ?

6. 6 / 31
❖ We proposed Videomark to help students catch
course concepts and review course easily.
❖ Videomark’s features:
1. Summary every weeks’ course contents into a
keyword cloud.
2. For each keyword in cloud, the system will
recommend you a set of video sections that most
likely to be the learning interest.
3. Automatically update data for analysis.

7. 7 / 31
User Interface

8. 8 / 31
User Interface

9. 9 / 31
User Interface

10. 10 / 31
User Interface

11. 11 / 31
What do we need ?
❖ To achieve our goal we need
▪ Students’ video activity records
▪ Lecture videos’ subtitle
❖ Students’ video activity records help as to locate
interests section in video.
❖ We use lecture videos’ subtitle to know the content
of video.

12. 12 / 31
Outline
❖ Introduction
❖ Implementation
❖ Experimental Results
❖ Conclusion
❖ Future Work

13. 13 / 31
Overview

14. 14 / 31
Web server and data service
…
Web server Data service server
user
Analysis system

15. 15 / 31
Analysis System (1/6)

16. 16 / 31
❖ Weighting the keyword
▪ We weighted the keyword with hot-segments we
mentioned previously.
▪ The hotter the video section is, the bigger the keyword
will be.
▪ The file combines with keyword and hot-section is
called hot-keyword file.
Analysis System (4/6)

17. 17 / 31
❖ Weighting the Keyword
Analysis System(5/6)

18. 18 / 31
Outline
❖ Introduction
❖ Implementation
❖ Experimental Results
❖ Conclusion
❖ Future Work

19. 19 / 31
❖ Network Security (2016-spring)
▪ Basic Information
Experimental Results
Course Duration 2016/5/15 – 2016/6/21
Chapter Count 7
Video Count 51
Student Number 2868
Pass Students 277
Pass Rate 9.66%

20. 20 / 31
❖ Learning Performance
Experimental Results
User learn with keyword cloud User learn without keyword cloud
Person 13 283
Average
Score
90.37 85.57
Pass Person 13 263
Pass Rate 100% 92.93%

21. 21 / 31
❖ Learning Engagement
Experimental Results
User learn with keyword cloud User learn without keyword cloud
Person 26 1636
Average
watch count
per week
53.14 16.14
Average
Forum Post
0.64 0.06
Average
Forum Reply
2.36 0.17
Average
Forum Like
5.16 0.36

22. 22 / 31
Learning Performance (Questionnaire)
strong
agree
agree no
comment
disagree strong
disagree
Keyword cloud makes me easier to
remember the content of course.
34.7 48.6 15.3 1.4 0
Every chapters’s keywords have
strong impression on me.
27.8 56.9 0 0 0
I can remember concepts related to
keywords much easier.
30.6 51.4 16.7 1.4 0
Keyword cloud makes me understand
the sketch of this course before
watching video.
29.2 48.6 19.4 0 0
Keyword cloud makes me much easier
to understand the points in course.
26.4 58.3 12.5 2.8 0
Keyword cloud helps me review the
course faster.
22.2 51.4 25 1.4 0
Keyword cloud helps me find answer of
questions I encounted in course faster.
16.7 43.1 29.2 11.1 0
Keyword cloud make me understand
more about Network Security in live.
25 43.1 26.4 5.6 0

23. 23 / 31
❖ Learning Engagement (Questionnaire)
Learning Engagement (Questionnaire)
strong
agree
agree no
comment
disagree strong
disagree
Keyword cloud make me desire to solve my
question.
11.1 62.5 19.4 6.9 0
Keyword cloud increase the time I use
ShareCourse.
11.1 25 48.6 15.3 0
I’ll watch sections repeatly that keyword cloud
labeled.
19.4 45.8 22.2 12.5 0
I’ll think more about concepts appeared in
keyword cloud.
16.7 66.7 13.9 2.8 0
I’ll focus on forum discussion about concepts in
keyword cloud.
23.6 44.4 26.4 5.6 0
Learning with keyword cloud make feel happy 13.9 50 30.6 5.6 0
I always looking forward to the new week
keyword cloud.
15.3 43.1 30.6 11.1 0
After seeing the keyword cloud in new week, I’ll
be more exciting to watch lecture video this week.
18.1 44.4 33.3 4.2 0
Keyword cloud make me more focus on learning. 16.4 30.6 31.9 20.8 0

24. 24 / 31
User Experience (Questionnaire)
strong agree agree no comment disagree strong
disagree
I can find button expand
keyword cloud easily.
25 51.4 15.3 6.9 1.4
The positon of expanded
keyword cloud is obvious to
see.
13.9 63.9 16.7 2.8 2.8
I can understand which
keyword is more important by
the size of them in keyword
cloud.
25 56.9 16.7 1.4 0
I think words in keyword
cloud are all important
concept
19.4 50 29.2 0 1.4

25. 25 / 31
❖ Any Problem when you use the KeywordCloud？
What can we do better？
▪ Sometime the word in keyword coloud is too small to click or overlay
with other words
▪ Some words in keyword cloud are too small that I often ignore them,
hope the word can be bigger.
▪ There should be a discription or animation to let user know hyperlinks
can link back to related video sections or make it clickalbe more
obviousy.
▪ Never used similar function on this website before, don’t know what the
button do, maybe add a information block to tell how the button work is
better.
Optional Question

26. 26 / 31
❖If we provide keyword cloud in the future, would you
use this function ? Why ?
▪ Yes, it provides me hints in course, fast search on video segment as well.
▪ Yes, It makes me see all the core concepts in course before I watch the video
which can make me aware of the direction and procedure I learn.
▪ Yes, my habit is to watch keyword cloud first and start watching the lecture
videos, this function help me focus on specific concepts in course and whenever I
have questions, I can find the review point in video easily by keyword cloud to
solve my question.
▪ Yes, I used ShareCourse last year, when I encountered some questions or
reviewed the course, it costed me lots of time to find the section talking about
the relative concept. Since I used keyword cloud, I feel it improve my efficiency a
lot.
Optional Question

27. 27 / 31
Outline
❖ Introduction
❖ Implementation
❖ Experimental Results
❖ Conclusion
❖ Future Work
❖ References

28. 28 / 31
Conclusion
❖ Videomark is valuable for learners to quick identify
the most important or difficult concepts in each topic.
❖ Help students to find specific video sections by
concept (indexing).
❖ Useful for the teacher to more understand which
parts of the contents can be further improved.
❖ Base on statistics and questionnaires we discover
that the keyword cloud has the tendency to improves
students’ learning engagement.

29. 29 / 31
Outline
❖ Introduction
❖ Implementation
❖ Experimental Results
❖ Conclusion
❖ Future Work

30. 30 / 31
❖ Tools can auto generate Chinese speaking lecture is
needed to be developed to break through the
limitation.
❖ The web page of Videomark is lack of information to
use it. Maybe we can add some animation or pop-up
description to help user make good use of Videomark.
❖We need a platform for teacher or TAs to manage
course elements. With this interface, teaching group
can upload files that Videomark needs and make it
easy to let every courses use Videomark service.
Future Work

31. 31 / 61
❖ [1] C. Yeager, B. Hurley-Dasgupta, and C. A. Bliss, “cmoocs and global
learning: An authentic alternative.” Journal of Asynchronous Learning
Networks, vol. 17, no. 2, pp. 133–147, 2013.
❖ [2] A. Ng and D. Koller, “Coursera,” Retrieved May 15, 2016, from the
World Wide Web: https://zh-tw.coursera.org/, 2012.
❖ [3] M. I. of Technology and H. University, “edx,” Retrieved May 15, 2016,
from the World Wide Web: https://www.edx.org/, 2012.
❖ [4] M. S. Sebastian Thrun, David Stavens, “Udacity,” Retrieved May 15,
2016, from the World Wide Web: https://www.udacity.com/, 2012.
❖ [5] L. Breslow, D. E. Pritchard, J. DeBoer, G. S. Stump, A. D. Ho, and D. T.
Seaton, “Studying learning in the worldwide classroom: Research into
edx’s first mooc,” Research & Practice in Assessment, vol. 8, 2013.
❖ [6] D. T. Seaton, Y. Bergner, I. Chuang, P. Mitros, and D. E. Pritchard, “Who
does what in a massive open online course?” Communications of the ACM,
vol. 57, no. 4, pp. 58–65, 2014.
Reference

32. 32 / 61
❖ [7] J. Kim, P. T. Nguyen, S. Weir, P. J. Guo, R. C. Miller, and K. Z. Gajos,
“Crowdsourcing step-by-step information extraction to enhance existing
how-to videos,” in Proceedings of the 32nd annual ACM conference on
Human factors in computing systems. ACM, 2014, pp. 4017–4026.
❖ [8] A. Agrawal, J. Venkatraman, S. Leonard, and A. Paepcke, “Youedu:
Addressing confusion in mooc discussion forums by recommending
instructional video clips,” 2015.
❖ [9] N. T. University, “Sharecourse,” Retrieved June 6, 2016, from the World
Wide Web: http://www.sharecourse.net/sharecourse/, 2012.
❖ [10] L. Pappano, “The year of the mooc,” The New York Times, vol. 2, no. 12,
p. 2012, 2012
❖ [11] T. University, “Xuetangx,” Retrieved June 2, 2016, from the World
Wide Web: http://www.xuetangx.com/, 2013. [12] N. Li, Ł. Kidzinski, P.
Jermann, and
Reference

33. 33 / 61
❖ [12] N. Li, Ł. Kidzinski, P. Jermann, and P. Dillenbourg, ´ MOOC Video
Interaction Patterns: What Do They Tell Us? Cham: Springer International
Publishing, 2015, pp. 197–210. [Online]. Available:
http://dx.doi.org/10.1007/ 978-3-319-24258-3 15
❖ [13] C. Shi, S. Fu, Q. Chen, and H. Qu, “Vismooc: Visualizing video
clickstream data from massive open online courses,” in 2015 IEEE Pacific
Visualization Symposium (PacificVis), April 2015, pp. 159–166
❖ [14] J. Kim, P. J. Guo, C. J. Cai, S.-W. D. Li, K. Z. Gajos, and R. C. Miller,
“Datadriven interaction techniques for improving navigation of
educational videos,” in Proceedings of the 27th annual ACM symposium on
User interface software and technology. ACM, 2014, pp. 563–572.
❖ [15] A. Leff and J. T. Rayfield, “Web-application development using the
model/view/controller design pattern,” in Enterprise Distributed Object
Computing Conference, 2001. EDOC ’01. Proceedings. Fifth IEEE
International, 2001, pp. 118–127.
Reference

34. 34 / 61
❖ [16] R. Dahl, “Node.js,” Retrieved June 6, 2016, from the World Wide Web:
https: //nodejs.org/en/, 2009.[13] C. Shi, S. Fu, Q. Chen, and H. Qu,
“Vismooc: Visualizing video clickstream data from massive open online
courses,” in 2015 IEEE Pacific Visualization Symposium (PacificVis), April
2015, pp. 159–166
❖ [17] D. Ritchie, “C,” Retrieved June 6, 2016, from the World Wide Web:
https://en. wikipedia.org/wiki/C (programming language), 1972.
❖ [18] B. Stroustrup, “C++,” Retrieved June 6, 2016, from the World Wide
Web: https: //isocpp.org/, 1995.
❖ [19] B. Eich, “Java script,” Retrieved June 6, 2016, from the World Wide
Web: https: //developer.mozilla.org/zh-TW/docs/Web/JavaScript, 1995.
❖ [20] M. Inc., “mongodb,” Retrieved June 6, 2016, from the World Wide Web:
https: //www.mongodb.com/, 2009.
❖ [21] G. Inc., “Google cloud platform,” Retrieved June 6, 2016, from the
World Wide Web: https://cloud.google.com/, 2011.
Reference

35. 35 / 61
❖ [22] Iron.io, “Ironworker,” Retrieved June 6, 2016, from the World Wide
Web: https: //www.iron.io/platform/ironworker/.
❖ [23] “Jieba,” Retrieved June 6, 2016, from the World Wide Web:
https://github.com/ fxsjy/jieba.
❖ [24] “d3-cloud.js,” Retrieved June 6, 2016, from the World Wide Web:
https://github. com/jasondavies/d3-cloud.
❖ [25] J. T. Mark Otto, “Bootstrap,” Retrieved June 6, 2016, from the World
Wide Web: http://getbootstrap.com/
❖ [26] J. Resig, “jquery,” Retrieved June 6, 2016, from the World Wide Web:
https:// jquery.com/, 2006.
Reference

36. 36 / 31

37. 37 / 31
❖ Counting the video seek event
▪ Use seek event in video to find out segments catch
users’ attention which are hot-segments.
▪ Choose at most five hot-segments each video.
Analysis System (2/6)

38. 38 / 31
❖ Subtitle Segmentation
▪ Filter stop-word eq.
this,that…
▪ Use word-frequency to
help segmentation.
▪ Instructor can provide
concept word file to help
keyword generation.
Analysis System (3/6)