Ph.D. in Telematics Engineering Defense - 31st of May, 2017 Title: Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments URL Thesis: http://eprints.networks.imdea.org/1582/1/ThesisJoseRuiperez_IMDEA.pdf YouTube video: https://www.youtube.com/watch?v=_k571iTNaCA Abstract: The 'big data' scene has brought new improvement opportunities to most products and services, including education. Web-based learning has become very widespread over the last decade, which in conjunction with the MOOC phenomenon, it has enabled the collection of large and rich data samples regarding the interaction of students with these educational online environments. We have detected different areas in the literature that still need improvement and more research studies. Particularly, in the context of MOOC and SPOC, where we focus our data analysis on the platforms Khan Academy, Open edX and Coursera. More specifically, we are going to work towards learning analytics visualization dashboards, carrying out an evaluation of these visual analytics tools. Additionally, we will delve into the activity and behavior of students with regular and optional activities, badges and their online academically dishonest conduct. The analysis of activity and behavior of students is divided first in exploratory analysis providing descriptive and inferential statistics, like correlations and group comparisons, as well as numerous visualizations that facilitate conveying understandable information. Second, we apply clustering analysis to find different profiles of students for different purposes e.g., to analyze potential adaptation of learning experiences and pedagogical implications. Third, we also provide three machine learning models, two of them to predict learning outcomes (learning gains and certificate accomplishment) and one to classify submissions as illicit or not. We also use these models to discuss about the importance of variables. Finally, we discuss our results in terms of the motivation of students, student profiling, instructional design, potential actuators and the evaluation of visual analytics dashboards providing different recommendations to improve future educational experiments.

1. Introduction Background and Method Results and Discussion Conclusions
Analyzing the Behavior of Students Regarding
Learning Activities, Badges, and Academic
Dishonesty in MOOC Environments
Author: Jos´e A. Ruip´erez Valiente
Advisor: Dr. Pedro J. Mu˜noz Merino
Ph.D. in Telematics Engineering
31st
of May 2017
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 1
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

2. Introduction Background and Method Results and Discussion Conclusions
Contents
1 Introduction
Motivations
Objectives
2 Background and Method
Related Work
Method
3 Results and Discussion
Exploratory Analysis
Clustering and Student Profiling
Analysis of Learning Outcomes based on Machine Learning Models
Evaluation of the Visual Analytics Tool ANALYSE
4 Conclusions
Final Remarks and Limitations
Future Work
Impact of the Thesis
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 2
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

3. Introduction Background and Method Results and Discussion Conclusions
Introduction
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 3
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

4. Introduction Background and Method Results and Discussion Conclusions
Introduction
The ‘big data’ scene has brought new opportunities to products
and services, including education
Web-based learning and Massive Open Online Courses
(MOOCs) have enhanced the collection of rich data samples
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 4
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

5. Introduction Background and Method Results and Discussion Conclusions
Motivations
Motivations
Learning analytics as an intersection between data science and
learning sciences [Gaˇsevi´c et al., 2017]
It’s a growing field and many areas need more work:
More thorough analysis of the behavior of students with
different learning activities and badges
Analysis and modeling of learning outcomes delving into the
importance of variables
Investigation of academic dishonesty behaviors in MOOCs
Evaluation of LA visualization dashboards for the specificities
of MOOC environments
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 5
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

6. Introduction Background and Method Results and Discussion Conclusions
Objectives
Objectives
Analysis of the following students’ aspects:
Interaction and behavior with regular and optional activities
Use and behavior of students with badges
Prediction of learning outcomes
Analyze academic dishonesty and illicit collaboration:
Design and implementation of algorithms to detect academic
dishonesty and its application to MOOC data
Analysis of students committing unethical behaviors, student
profiling and instructional design guidelines
Evaluation of the usability, usefulness and effectiveness of a LA
visualization dashboard for MOOCs
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 6
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

7. Introduction Background and Method Results and Discussion Conclusions
Background and Method
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 7
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

8. Introduction Background and Method Results and Discussion Conclusions
Related Work
MOOCs and SPOCs
MOOCs are the new stage in online education [Masters, 2011]
Instructors cannot monitor all students’ actions anymore
Video lectures and automatic graded assignments [Voss, 2013]
Communication tools and social activity [Nicoar˘a, 2013]
SPOCs use similar technologies, but the number of users is
smaller and the access is restricted [Fox, 2013]
We analyze data from both MOOCs and SPOCs
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 8
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

9. Introduction Background and Method Results and Discussion Conclusions
Related Work
Learning Analytics and Educational Data Mining
Both fields reflect the importance of analyzing data in
education [Siemens and Baker, 2012]
EDM is more focused in automatic discovery and modeling
LA usually focuses on informing and empowering instructors
Studies can be divided in different areas according to various
factors [Papamitsiou and Economides, 2014]
Learning environment, context and specific settings
The objectives and goals
The actual methods that are applied
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 9
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

10. Introduction Background and Method Results and Discussion Conclusions
Related Work
LA Dashboards and its Evaluation I
Visual analytics is a common technique to transfer information
There are a number of LA dashboards in the literature e.g.,
GISMO for Moodle [Mazza and Milani, 2005], CourseVis
[Mazza and Dimitrova, 2004] for WebCT or PeakVizor
[Chen et al., 2016] for data of Coursera and Open edX,
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 10
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

11. Introduction Background and Method Results and Discussion Conclusions
Related Work
LA Dashboards and its Evaluation II
At UC3M we have contributed with ALAS-KA
[Ruip´erez-Valiente et al., 2015] for Khan Academy and
ANALYSE for Open edX [Ruip´erez-Valiente et al., 2016b]
Studies are required to evaluate the usability of these tools e.g.,
[Charleer et al., 2014, Govaerts et al., 2012]
We contribute with the evaluation of ANALYSE
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 11
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

12. Introduction Background and Method Results and Discussion Conclusions
Related Work
Analysis of Learning Outcomes
The prediction of learning outcomes is common in education:
Diverse goals e.g., to predict dropouts [Kloft et al., 2014] or a
score of a test [Pardos et al., 2010]
Different methods from simple linear regressions
[Grafsgaard et al., 2014] to complex neuronal networks
[Calvo-Flores et al., 2006]
Introduce findings in a tool to improve the learning process e.g.,
Student Success System (S3) [Essa and Ayad, 2012]
Our contribution in the area is the prediction of learning gains
and certificate accomplishment analyzing variable importance
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 12
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

13. Introduction Background and Method Results and Discussion Conclusions
Related Work
Gamification and Use of Badges
Gamification applies game components in non-game contexts to
increase motivation and engagement [Deterding et al., 2011]
Positive outcomes depending on context [Hamari et al., 2014]
Not always positively perceived [Berkling and Thomas, 2013]
We focus on badges [Goligoski, 2012]
Positive studies in education to increment frequency of visits
[Muntean, 2011] and social activity [Barata et al., 2013]
Our contribution is to explore and analyze the behavior of
students with badges with novel metrics and correlations
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 13
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

14. Introduction Background and Method Results and Discussion Conclusions
Related Work
Academic Dishonesty and Illicit Collaboration
“Any type of fraudulent action in an academic work”
[Lambert et al., 2003]
Affected by factors like demographics [Harding et al., 2007],
technology and online learning
We focus in the area of cheating in MOOCs:
Students sign a code of honor when creating their account
We contribute with novel algorithms to detect CAMEO and
‘close submitters’, exploring and discussing their results
Related to ‘gaming the system’ [Baker et al., 2008]
Other studies inspected CAMEO [Northcutt et al., 2016]
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 14
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

15. Introduction Background and Method Results and Discussion Conclusions
Method
Overview of the Method
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 15
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

16. Introduction Background and Method Results and Discussion Conclusions
Method
MOOC Platforms
Khan Academy: Non-for-profit founded by Salman Khan
EdX: Non-for-profit founded by MIT and Harvard
Open source code as the collaborative project Open edX
Coursera: For-profit founded by two Stanford professors
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 16
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

17. Introduction Background and Method Results and Discussion Conclusions
Method
Optional Activities in Khan Academy
Avatar and display badges
Feedback and votes Goals
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 17
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

18. Introduction Background and Method Results and Discussion Conclusions
Method
Badges in Khan Academy
We focus on the following two categories:
Topic badges: These badges are awarded to students when
they accomplish to earn proficiency in a set of exercises (skills)
Can be earned only once per student
Repetitive badges: Can be earned repetitively by the same
student by fulfilling certain criteria
‘Timed Problem’ badges for solving problems rapidly
‘Streak’ badges for solving several exercises correctly in a row
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 18
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

19. Introduction Background and Method Results and Discussion Conclusions
Method
Example of Badges in Khan Academy
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 19
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

20. Introduction Background and Method Results and Discussion Conclusions
Method
Learning Analytics Visualization Dashboards
ALAS-KA: Visual analytics tool for Khan Academy
ANALYSE: Visual analytics tool for Open edX
ALAS-KA ANALYSE
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 20
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

21. Introduction Background and Method Results and Discussion Conclusions
Method
Case Studies: SPOCs with Khan Academy
Remedial courses on math, chemistry and physics taken by first-year
students at UC3M.
Summer 2013
A total number of 67 students in the physics course, 73
students in chemistry and 243 students in mathematics
Summer 2014
Pre-test and post-test to compute learning gains
(LG = posttest − pretest)
156 students for physics and 69 for chemistry
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 21
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

22. Introduction Background and Method Results and Discussion Conclusions
Method
Case Studies: MOOCs on Coursera and edX
MOOC ‘The Spain of Don Quixote’ by UAM on edX
3530 enrollments and 164 certificates issued
MOOC ‘Mechanics Review’ by MIT on edX
13500 enrollments and 502 certificates issued
About 1000 problems and 69 videos
MOOCs ‘Music Theory’ and ‘Introduction to Philosophy’ by the
University of Edinburgh on Coursera
Music was 5 weeks long with 10–14 questions per week, 5.159
accounts submitted all graded assignments
Philosophy was 7 weeks long with 6–12 questions per week,
2359 accounts submitted all graded assignments
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 22
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

23. Introduction Background and Method Results and Discussion Conclusions
Method
Selected indicators: General
Some indicators might make sense only in some platforms and not
all case studies use all indicators. Divided as follows:
Use of the platform e.g., exercises accessed or active days
Correct progress in the platform e.g., completed videos or
performance in first attempt
Time in the platform e.g., exercise or page time
Problem and submission e.g., type of assignment or location
Behavior solving exercises e.g., hint abuse or video avoidance
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 23
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

24. Introduction Background and Method Results and Discussion Conclusions
Method
Selected indicators: Behavior with badges
Intentionality on topic badges: It infers if a student is trying to
maximize the number of topic badges achieved
Intentionality on repetitive badges: It analyzes if students are
purposely earning repetitive badges, instead of as part of the
learning process
Concentration on achieving badges: It measures if the
consecutive actions of a student are devoted to fulfill the
requirements of a badge
Time efficiency in badges: total number of badges divided by
the total time in the platform
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 24
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

25. Introduction Background and Method Results and Discussion Conclusions
Method
Selected indicators: Close submitters
Detect user accounts of students in online courses that always
submit their assignments very close in time
SP =





sp11 sp12 . . . sp1M
sp21 sp22 . . . sp2M
...
...
...
...
spN1 spN2 . . . spNM





D =





d11 d12 . . . d1N
d21 d22 . . . d2N
...
...
...
...
dN1 dN2 . . . dNN





dMAD
ij =
1
M
M
k=1 |spik − spjk|
We establish a distance threshold and accounts below that value are
denominated as close submitters
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 25
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

26. Introduction Background and Method Results and Discussion Conclusions
Method
Selected indicators: CAMEO
Stands for Copying Answers using Multiple Existences Online
Student gets the solution with one or more harvesting accounts
(the harvester/s), and then submits it in the master account
They use ‘show answer’ or just ‘exhaustive search’
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 26
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

27. Introduction Background and Method Results and Discussion Conclusions
Results and Discussion
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 27
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

28. Introduction Background and Method Results and Discussion Conclusions
Exploratory Analysis
Use of Regular and Optional Activities
Good use of regular activities, low for optional activities
Only 23.2% of the students used at least one of the activities
The activities more used are not related to learning
Optional activity Feedback Vote Goal Profile avatar Badge display
Percentage 4.1% 6.6% 6.2% 10.8% 12%
Type of activity Percentage of activities accessed
0% 1-33% 34-66% 67-99% 100%
Regular learning activities 2.48% 51.55% 23.19% 18.84% 3.93%
Optional activities 76.81% 18.43% 4.14% 0.41% 0.21%
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 28
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

29. Introduction Background and Method Results and Discussion Conclusions
Exploratory Analysis
Relationship of Optional Activities and Learning
Outcomes
Metric
Optional
Activities
Goal Feedback Vote Avatar
Display
badges
Pearson correlation
Proficient exercises
0.553** 0.384** 0.205** 0.243** 0.415** 0.418**
Partial correlation
Proficient exercises
0.282** 0.25** -0.04 -0.031 0.235** 0.229**
Pearson correlation
Learning gain
0.293** 0.102 0.219 0.333* 0.221 0.296**
Partial correlation
Learning gain
0.142 -0.07 0.124 0.214 0.17 0.261*
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 29
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

30. Introduction Background and Method Results and Discussion Conclusions
Exploratory Analysis
Overview of Badge Activity
Some students achieved really high (>500) amounts of badges
Different degrees of interest in badges
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 30
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

31. Introduction Background and Method Results and Discussion Conclusions
Exploratory Analysis
Overview of Close Submitters Detected
We apply the close submitter algorithm, MAD threshold 35 min
99 accounts in music and 26 in philosophy
Mostly grouped in couples but also bigger communities
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 31
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

32. Introduction Background and Method Results and Discussion Conclusions
Exploratory Analysis
Close Submitters vs. Rest of Accounts
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 32
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

33. Introduction Background and Method Results and Discussion Conclusions
Exploratory Analysis
Overview of the Amount of CAMEO
A significant percentage uses CAMEO to get a certificate
3.7% of the certificate earners used CAMEO to obtain more
than 50% of their correct answers
#Master
accounts
#Harvester
accounts
#Harvested
answers
Certificate
earners
65 (12.9%) 78 17350 (4.3%)
Non-certificate
earners
84 (7.7%) 74 12438 (5.1%)
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 33
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

34. Introduction Background and Method Results and Discussion Conclusions
Exploratory Analysis
CAMEO Accounts vs. Rest of Accounts
Time and performance scatterplot Performance distribution
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 34
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

35. Introduction Background and Method Results and Discussion Conclusions
Exploratory Analysis
Distribution of CAMEO Over Course Timeline
Motivation appears to be the certificate
There are different profiles of CAMEO users
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 35
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

36. Introduction Background and Method Results and Discussion Conclusions
Exploratory Analysis
Factors Associated with CAMEO
High-stake questions: CAMEO prevalence of 7.25%, 5.65%
and 5.09% for quizzes, homework and checkpoints respectively
Delayed feedback: Delaying ‘show answer’ feedback
decreases approximately the amount of CAMEO by half
Randomized variables: Problems containing statements with
randomized variables have half the amount of CAMEO
compared to those without randomized variables (p < 0.01)
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 36
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

37. Introduction Background and Method Results and Discussion Conclusions
Clustering and Student Profiling
Overview of the Clustering and Student Profiling
We apply Two-Step clustering to group students by their
behavior:
1 Behavior with regular and optional activities
2 Behavior with badges
3 Illicit collaborations and academic dishonesty
Clusters show different profiles of students
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 37
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

38. Introduction Background and Method Results and Discussion Conclusions
Clustering and Student Profiling
Clustering by Regular and Optional Activities
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 38
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

39. Introduction Background and Method Results and Discussion Conclusions
Clustering and Student Profiling
Clustering by Behavior with Badges
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 39
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

40. Introduction Background and Method Results and Discussion Conclusions
Clustering and Student Profiling
Clustering by the Type of Illicit Collaborations
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 40
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

41. Introduction Background and Method Results and Discussion Conclusions
Clustering and Student Profiling
Types of Couples and Communities
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 41
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

42. Introduction Background and Method Results and Discussion Conclusions
Analysis of Learning Outcomes based on Machine Learning Models
Overview of the Machine Learning Models
This subsection provides three machine learning models:
1 Prediction of learning gains
2 Prediction of certificate accomplishment
3 Classification of submissions as CAMEO or not
For each model we describe the methodology, training process
and evaluation of the model
We discuss and interpret the results that we have obtained
taking into account the variable importance
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 42
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

43. Introduction Background and Method Results and Discussion Conclusions
Analysis of Learning Outcomes based on Machine Learning Models
Prediction of Learning Gains – Method
Dataset: We use the SPOCs on physics and chemistry that
have available a pre-test and post-test
Algorithm: We use a multivariate linear regression as we
expect a linear relationship
Variable selection: Stepwise procedure and literature
Variable importance: We use the standardized coefficients of
the regression model
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 43
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

44. Introduction Background and Method Results and Discussion Conclusions
Analysis of Learning Outcomes based on Machine Learning Models
Prediction of Learning Gains – Evaluation
Model Independent Variable
Un-std. Coeff. Std. Coeff.
B Std. Error Beta
Final
Constant 13.615 9.734
pre test score - 0.668 0.071 - 0.727
avg attempts 6.426 3.142 0.187
exercise effectiveness no help 0.392 0.104 0.324
avg daytime 0.824 0.230 0.342
total abandonment 0.143 0.097 0.155
negative behaviors - 0.721 0.223 - 0.264
R R Squared Std. Error of the Prediction
0.825 0.68 13.3
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 44
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

45. Introduction Background and Method Results and Discussion Conclusions
Analysis of Learning Outcomes based on Machine Learning Models
Prediction of Certificate Accomplishment –
Method
Dataset: ‘The Quixote’ MOOC on edX
Algorithm: We test the performance of RF, LG, GBM and
kNN for early prediction
Variable importance: We compute the relative variable
importance for GBM as reported [Friedman, 2001]
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 45
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

46. Introduction Background and Method Results and Discussion Conclusions
Analysis of Learning Outcomes based on Machine Learning Models
Prediction of Certificate Accomplishment –
Evaluation of the Models
Based on the comparison we select the GBM model
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 46
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

47. Introduction Background and Method Results and Discussion Conclusions
Analysis of Learning Outcomes based on Machine Learning Models
Prediction of Certificate Accomplishment –
Variable Importance
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 47
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

48. Introduction Background and Method Results and Discussion Conclusions
Analysis of Learning Outcomes based on Machine Learning Models
Classification of CAMEO Submissions – Method
Dataset: ‘Mechanics Review’ MOOC on edX
Algorithm: We select a RF model since its useful to rank the
importance of variables
Variable selection and importance: We use VSURF
[Genuer et al., 2015] reporting the out-of-bag error
[Breiman, 2001]:
VI(Xj
) =
1
ntree
ntree
t
(errOOB
j
t − errOOBt)
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 48
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

49. Introduction Background and Method Results and Discussion Conclusions
Analysis of Learning Outcomes based on Machine Learning Models
Classification of CAMEO Submissions – Evaluation
Confusion matrix:
Classification
Reference
Non-CAMEO CAMEO
Non-CAMEO 93.852% 0.194%
CAMEO 0.366% 5.588%
Evaluation metrics:
Metric AUC Sensitivity Specificity
Kappa
coefficient
Accuracy
Baseline
accuracy
Value 0.9993 0.9664 0.99611 0.9493 0.9944 0.9421
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 49
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

50. Introduction Background and Method Results and Discussion Conclusions
Analysis of Learning Outcomes based on Machine Learning Models
Classification of CAMEO Submissions – Variable
Importance
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 50
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

51. Introduction Background and Method Results and Discussion Conclusions
Evaluation of the Visual Analytics Tool ANALYSE
Overview of ANALYSE
12 visualizations divided in problem (3), video (4) and course
activity (5)
Each visualization has a description, graph and legend
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 51
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

52. Introduction Background and Method Results and Discussion Conclusions
Evaluation of the Visual Analytics Tool ANALYSE
Interacting with ANALYSE
Fictitious data but resembling a real case study
They can interact as instructors with the graphs while also
selecting different students and learning resources
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 52
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

53. Introduction Background and Method Results and Discussion Conclusions
Evaluation of the Visual Analytics Tool ANALYSE
Methodology of the Survey
A 60 minutes intervention with 40 respondents with 6 phases:
1 Initial interaction with a typical course using Open edX
2 Initial interaction with ANALYSE
3 Respondents interacted with the 12 visualizations and had to
respond a question by interacting with each graph
4 Respondents had to rate usefulness of each visualization and
ANALYSE globally as a tool
5 Respondents were asked the 10 questions of the SUS survey
[Brooke, 1996]
6 Finally, the respondents received two open questions
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 53
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

54. Introduction Background and Method Results and Discussion Conclusions
Evaluation of the Visual Analytics Tool ANALYSE
Evaluation of the Visual Analytics Tool ANALYSE
The effectiveness was high with all questions above 90%
correctness except for one problematic question (around 30%)
Good average value of visualizations usefulness 3.69/5 with big
differences between some visualizations and 4.2/5 globally
The SUS questionnaire obtained an score of 78.4 which is
within the 15% best percentile [Sauro, 2011]
LARAe system [Charleer et al., 2014] with an SUS score of 76
or the SAM tool [Govaerts et al., 2012] with a score of 71.36
Two open questions with comments and recommendations e.g.,
“I knew everything had happened” or “The app was very
intuitive and I did not need any previous knowledge”
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 54
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

55. Introduction Background and Method Results and Discussion Conclusions
Conclusions
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 55
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

56. Introduction Background and Method Results and Discussion Conclusions
Final Remarks and Limitations
Final Remarks and Limitations I
Analysis of behavior with regular and optional activities:
Different profiles for adaptation purposes
Relationship between optional activities and learning outcomes
Potential actuators based on the prediction models
Models based on a single course and tested retrospectively
Analysis of the behavior of students with badges:
Different degrees of interest and behavioral profiles
Some student showed very high intentionality for repetitive
badges which might be counterproductive for learning
These results might be tied to the specificities of our
educational context
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 56
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

57. Introduction Background and Method Results and Discussion Conclusions
Final Remarks and Limitations
Final Remarks and Limitations II
Analysis of academic dishonestly based on 2 novel algorithms:
Potential run-time detection based on machine learning
Academic dishonesty is a significant issue in online education:
1 It diminishes the value of MOOC certificates
2 Cheating is usually associated with poor learning
3 Interferes with educational research
Address generalization by analyzing a larger MOOC portfolio
No confirmation regarding students’ cheating behavior
We also conducted an extensive evaluation for ANALYSE
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 57
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

58. Introduction Background and Method Results and Discussion Conclusions
Future Work
Future Work
Multiple potential next steps, some of them are as follow:
Based on the predictors and detectors integrate our findings as
actuator systems in ALAS-KA and ANALYSE
Develop models that can generalize well across different courses
and contexts
Conduct A/B experiments to more effectively analyze the effect
of optional activities and badges on students’ behavior
Measure more accurately the impact of the different illicit
behaviors on learning achievement
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 58
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

59. Introduction Background and Method Results and Discussion Conclusions
Impact of the Thesis
JCR Publications Based on Thesis Results
1 Mu˜noz-Merino, P. J., Ruip´erez-Valiente, J. A., Alario-Hoyos, C., P´erez-Sanagust´ın, M.,
& Delgado Kloos, C. (2015). Precise Effectiveness Strategy for analyzing the
effectiveness of students with educational resources and activities in MOOCs.
Computers in Human Behavior, 47, 108–118
2 Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., Delgado Kloos, C., Niemann, K.,
Scheffel, M., & Wolpers, M. (2016). Analyzing the Impact of Using Optional Activities
in Self-Regulated Learning. IEEE Transactions on Learning Technologies, 9(3), 231–243
3 Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., Pijeira D´ıaz, H. J., Santofimia Ruiz, J.,
& Delgado Kloos, C. (2017). Evaluation of a Learning Analytics Application for Open
edX Platform. Computer Science and Information Systems, 14(1), 51–73
4 Alexandron, G., Ruip´erez-Valiente, J. A., Chen, Z., Pedro J. Mu˜noz-Merino, &
Pritchard, D. E. (2017). Copying@Scale: Using Harvesting Accounts for Collecting
Correct Answers in a MOOC. Computers & Education, 108, 96–114
5 Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., & Delgado Kloos, C. (2017). Detecting
and Clustering Students by their Gamification Behavior with Badges: A Case Study in
Engineering Education (In press). The International Journal of Engineering Education
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 59
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

60. Introduction Background and Method Results and Discussion Conclusions
Impact of the Thesis
Conference Publications Based on Thesis Results
1 Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., Delgado Kloos, C., Niemann, K., &
Scheffel, M. (2014). Do Optional Activities Matter in Virtual Learning Environments?
In European Conference on Technology Enhanced Learning
2 Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., & Delgado Kloos, C. (2015). A
Predictive Model of Learning Gains for a Video and Exercise Intensive Learning
Environment. In International Conference on Artificial Intelligence in Education
3 Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., & Delgado Kloos, C. (2016). An
analysis of the use of badges in an educational experiment. In Frontiers in Education
4 Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., & Delgado Kloos, C. (2016). Analyzing
students’ intentionality towards badges within a case study using Khan academy. In
International Conference on Learning Analytics & Knowledge
5 Ruip´erez-Valiente, J. A., Alexandron, G., Chen, Z., & Pritchard, D. E. (2016). Using
multiple accounts for harvesting solutions in MOOCs. In Learning@Scale
6 Ruip´erez-Valiente, J. A., Joksimovi´c, S., Kovanovi´c, V., Gaˇsevi´c, D., Mu˜noz-Merino, P.
J., & Delgado Kloos, C. (2017). A Data-driven Method for the Detection of Close
Submitters in Online Learning Environments. In Conference on World Wide Web
7 Ruip´erez-Valiente, J. A., Cobos, R., Mu˜noz-Merino, P. J., And´ujar, A., & Delgado
Kloos, C. (2017). Early Prediction and Variable Importance of Certificate
Accomplishment in a MOOC. In European MOOCs Stakeholders Summit.
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 60
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

61. Introduction Background and Method Results and Discussion Conclusions
Impact of the Thesis
Other Publications
5 more JCR publications related to thesis topics
4 more conference publications related to thesis topics
2 JCR publications based on thesis results under review
1 Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., Alexandron, G.,
and Pritchard, D. E. Using Machine Learning to Detect
“Multiple-Account” Cheating and Analyze the Influence of Student
and Problem Features (Major revision). IEEE Transactions on
Learning Technologies
2 Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., and Delgado Kloos,
C. Improving the Prediction of Learning Outcomes in Educational
Platforms including Higher Level Interaction Indicators (Under
review). Expert Systems
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 61
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

62. Introduction Background and Method Results and Discussion Conclusions
Impact of the Thesis
Research Projects
Part of the results have been transferred to the following projects:
1 Educational Reflected Spaces (EEE)
2 eMadrid
3 Reformulate Scalable Educational Ecosystems Offering
Technological Innovations (RESET)
4 Supporting Higher Education to Integrate Learning Analytics
(SHEILA)
5 Spanish Network of Learning Analytics (SNOLA):
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 62
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

63. Introduction Background and Method Results and Discussion Conclusions
Impact of the Thesis
Research Stays
1 Three months at the Physics department of MIT where the
author joined RELATE group and was supervised by Prof.
David E. Pritchard. As a result one workshop presentation
[Alexandron et al., 2015] and two articles
[Ruip´erez-Valiente et al., 2016a, Alexandron et al., 2017] have
been published already
2 Three months at the School of Informatics at the University of
Edinburgh where the author joined the Institute for Adaptive
and Neural Computation and was supervised by Prof. Dragan
Gaˇsevi´c. We have already one paper published
[Ruip´erez-Valiente et al., 2017]
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 63
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

64. Introduction Background and Method Results and Discussion Conclusions
Impact of the Thesis
Other Merits
Our study on optional activities [Ruip´erez-Valiente et al., 2014]
was selected as one of the best 5 papers at the EC-TEL 2014
Runner up to the best demonstration paper award at the
EC-TEL 2015 [Pijeira D´ıaz et al., 2015]
Collaborator at the best pedagogical innovation project at
UC3M for the use of an Open edX instance with ANALYSE in a
high school for adult education
Our work on CAMEO [Ruip´erez-Valiente et al., 2016a] was
selected as one of the best papers at Learning@Scale
Conference 2016 receiving an ‘Honorable Mention’
Jos´e A. Ruip´erez Valiente – @JoseARuiperez 64
Analyzing the Behavior of Students Regarding Learning Activities, Badges, and Academic Dishonesty in MOOC Environments

65. Thank you for your patience!
Questions?

66. References I
Alexandron, G., Ruip´erez-Valiente, J. A., Chen, Z., Mu˜noz-Merino, P. J., and Pritchard,
D. E. (2017). Copying@Scale: Using harvesting accounts for collecting correct answers in a
MOOC. Computers & Education, 108:96–114.
Alexandron, G., Ruip´erez-Valiente, J. A., and Pritchard, D. E. (2015). Evidence of MOOC
Students Using Multiple Accounts to Harvest Correct Answers. In Learning with MOOCs II:
A Workshop for Practitioners: New Approaches to Teaching & Learning.
Baker, R. S., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., and Koedinger, K. (2008).
Why students engage in “gaming the system” behavior in interactive learning environments.
Journal of Interactive Learning Research, 19(2):185.
Barata, G., Gama, S., Jorge, J., and Gon¸calves, D. (2013). Engaging engineering students
with gamification. In 5th International Conference on Games and Virtual Worlds for Serious
Applications (VS-GAMES), pages 1–8. IEEE.
Berkling, K. and Thomas, C. (2013). Gamification of a Software Engineering course and a
detailed analysis of the factors that lead to it’s failure. In International Conference on
Interactive Collaborative Learning (ICL), pages 525–530. IEEE.
Breiman, L. (2001). Random forests. Machine learning, 45(1):5–32.
Brooke, J. (1996). SUS - A quick and dirty usability scale. Usability evaluation in industry,
189(194):4–7.

67. References II
Calvo-Flores, M. D., Galindo, E. G., Jim´enez, M. P., and Pi˜neiro, O. P. (2006). Predicting
students’ marks from moodle logs using neural network models. Current Developments in
Technology-Assisted Education, 1:586–590.
Charleer, S., Santos, J. L., Klerkx, J., and Duval, E. (2014). Improving teacher awareness
through activity, badge and content visualizations. In International Conference on
Web-Based Learning, pages 143–152. Springer.
Chen, Q., Chen, Y., Liu, D., Shi, C., Wu, Y., and Qu, H. (2016). Peakvizor: Visual analytics
of peaks in video clickstreams from massive open online courses. IEEE transactions on
visualization and computer graphics, 22(10):2315–2330.
Deterding, S., Dixon, D., Khaled, R., and Nacke, L. (2011). From game design elements to
gamefulness: defining gamification. In Proceedings of the 15th international academic
MindTrek conference: Envisioning future media environments, pages 9–15. ACM.
Essa, A. and Ayad, H. (2012). Improving student success using predictive models and data
visualisations. Research in Learning Technology, 20.
Fox, A. (2013). From MOOCs to SPOCs. Communications of the ACM, 56(12):38–40.
Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine.
Annals of statistics, pages 1189–1232.

68. References III
Gaˇsevi´c, D., Dawson, S., and Siemens, G. (2015). Let’s not forget: Learning analytics are
about learning. TechTrends, 59(1):64–71.
Gaˇsevi´c, D., Kovanovi´c, V., and Joksimovi´c, S. (2017). Piecing the Learning Analytics
Puzzle: A Consolidated Model of a Field of Research and Practice.
Genuer, R., Poggi, J.-M., and Tuleau-Malot, C. (2015). VSURF: An R Package for Variable
Selection Using Random Forests. The R Journal, 7(2):19–33.
Goligoski, E. (2012). Motivating the learner: Mozilla’s open badges program. Access to
Knowledge: A Course Journal, 4(1).
Govaerts, S., Verbert, K., Duval, E., and Pardo, A. (2012). The student activity meter for
awareness and self-reflection. In Human Factors in Computing Systems, pages 869–884.
ACM.
Grafsgaard, J., Wiggins, J., and Boyer, K. (2014). Predicting Learning and Affect from
Multimodal Data Streams in Task-Oriented Tutorial Dialogue. In Stamper, J., Pardos, Z.,
Mavrikis, M., and McLaren, B., editors, Proceedings of the 7th International Conference on
Educational Data Mining, pages 122–129.
Hamari, J., Koivisto, J., and Sarsa, H. (2014). Does gamification work? – a literature review
of empirical studies on gamification. In 47th International Conference on System Sciences
(HICSS), pages 3025–3034. IEEE.

69. References IV
Harding, T. S., Mayhew, M. J., Finelli, C. J., and Carpenter, D. D. (2007). The theory of
planned behavior as a model of academic dishonesty in engineering and humanities
undergraduates. Ethics & Behavior, 17(3):255–279.
Kloft, M., Stiehler, F., Zheng, Z., and Pinkwart, N. (2014). Predicting MOOC dropout over
weeks using machine learning methods. In Proceedings of the Workshop on Analysis of Large
Scale Social Interaction in MOOCs, pages 60–65.
Lambert, E. G., Hogan, N. L., and Barton, S. M. (2003). Collegiate academic dishonesty
revisited: What have they done, how often have they done it, who does it, and why did they
do it. Electronic Journal of Sociology, 7(4):1–27.
Masters, K. (2011). A Brief Guide To Understanding MOOCs. The Internet Journal of
Medical Education, 1(2):1–5.
Mazza, R. and Dimitrova, V. (2004). Visualising student tracking data to support instructors
in web-based distance education. In Proceedings of the 13th international World Wide Web
conference on Alternate track papers & posters, pages 154–161. ACM.
Mazza, R. and Milani, C. (2005). Exploring usage analysis in learning systems: Gaining
insights from visualisations. In Workshop on usage analysis in learning systems at 12th
international conference on artificial intelligence in education, pages 65–72.

70. References V
Muntean, C. I. (2011). Raising engagement in e-learning through gamification. In
Proceedings of the 6th International Conference on Virtual Learning, pages 323–329.
Nicoar˘a, E. S. (2013). The impact of massive online open courses in academic environments.
In Conference Proceedings of eLearning and Software for Education (eLSE), pages 644–649.
Universitatea Nationala de Aparare Carol I.
Northcutt, C. G., Ho, A. D., and Chuang, I. L. (2016). Detecting and preventing
“multiple-account” cheating in massive open online courses. Computers & Education,
100:71–80.
Papamitsiou, Z. and Economides, A. A. (2014). Learning Analytics and Educational Data
Mining in Practice: A Systematic Literature Review of Empirical Evidence. Educational
Technology & Society, 17(4):49–64.
Pardos, Z., Gowda, S., Baker, R. S., and Heffernan, N. (2010). Ensembling predictions of
student post-test scores for an intelligent tutoring system. In Educational Data Mining 2011.
Pijeira D´ıaz, H. J., Santofimia, J., Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., and
Delgado Kloos, C. (2015). Using Video Visualizations in Open edX to Understand Learning
Interactions of Students. In 10th European Conference on Technology Enhanced Learning,
EC-TEL 2015, Toledo, Spain, pages 522–525.

71. References VI
Ruip´erez-Valiente, J. A., Alexandron, G., Chen, Z., and Pritchard, D. E. (2016a). Using
Multiple Accounts for Harvesting Solutions in MOOCs. In Proceedings of the Third (2016)
ACM Conference on Learning@Scale, pages 63–70. ACM.
Ruip´erez-Valiente, J. A., Joksimovi´c, S., Kovanovi´c, V., Gaˇsevi´c, D., Mu˜noz-Merino, P. J.,
and Delgado Kloos, C. (2017). A Data-driven Method for the Detection of Close Submitters
in Online Learning Environments. In Proceedings of the 26th International Conference on
World Wide Web Companion, pages 361–368.
Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., Delgado Kloos, C., Niemann, K., and
Scheffel, M. (2014). Do Optional Activities Matter in Virtual Learning Environments? In
Ninth European Conference on Technology Enhanced Learning, pages 331–344, Graz,
Austria. Springer International Publishing.
Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., Gasc´on-Pinedo, J. A., and Delgado Kloos, C.
(2016b). Scaling to Massiveness with ANALYSE: A Learning Analytics Tool for Open edX
(In press). IEEE Transactions on Human-Machine Systems, pages 1–6.
Ruip´erez-Valiente, J. A., Mu˜noz-Merino, P. J., Leony, D., and Delgado Kloos, C. (2015).
ALAS-KA: A learning analytics extension for better understanding the learning process in the
Khan Academy platform. Computers in Human Behavior, 47:139–148.

72. References VII
Salehi, M. and Kamalabadi, I. N. (2013). Hybrid recommendation approach for learning
material based on sequential pattern of the accessed material and the learner’s preference
tree. Knowledge-Based Systems, 48:57–69.
Sauro, J. (2011). SUSTisfied? little-known System Usability Scale facts. UX Magazine,
10(3):2011–3.
Siemens, G. and Baker, R. S. (2012). Learning analytics and educational data mining:
towards communication and collaboration. In Proceedings of the 2nd international
conference on learning analytics and knowledge, pages 252–254. ACM.
Voss, B. D. (2013). Massive open online courses (MOOCs): A primer for university and
college board members. AGB Association of Governing Boards of Universities and Colleges.