The document discusses learning analytics research at UC3M. It describes 5 developments: 1) The CCOLAB system captures raw learning events in a virtual environment. 2) The LearnGlass platform facilitates creating visualizations of captured data. 3) The Genghis project analyzes online student behavior data from the Khan Academy platform. 4) The ClassON system uses analytics to provide just-in-time feedback to teachers and students in classrooms. 5) The ISCARE system applies analytics and gamification to motivate students through competitive exercises.

1. Learning Analytics@UC3M
Carlos Delgado Kloos, Abelardo Pardo (U. Sydney),
Pedro J. Muñoz-Merino, Israel Gutiérrez, Derick Leony
Universidad Carlos III de Madrid
www.it.uc3m.es/cdk
www.emadridnet.org

2. Education?
 Just knowledge transfer?
 One way process?
Teaching
K K’>K
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

3. Feedback!
 Feedback is needed to
 Control that learning occurs
 Take adequate measures if not
K K’
K’>K?
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

4. Assessment
 Simplest form of feedback
 Formative: to support learning
 Summative: also grading function
Teaching
Assessment
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

5. Analytics
 Another form of feedback
 Recollection of educational information
 Can be done especially well in the digital world
Teaching
Analytics
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

6. 5 Developments
 1. Capturing events (CCOLAB)
 2. Visualization (LearnGlass)
 3. Abstracting out data (Genghis)
 4. Just-in-Time teaching (ClassON)
 5. Gamification (ISCARE)
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

7. 1. Capturing Events
 CCOLAB system
 Capture raw events generated
 Use virtual machine
Capture
of data
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

8. Virtualizing the environment
Data
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

9. Virtual Machine
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

10. Events captured
 Power up and shutdown of virtual machine
 Commands used in the command line interface (bash)
 Execution of the compiler and errors obtained (gcc)
 Execution of the debugger and commands used (gdb)
 Execution of the editor and IDE (kate & kdevelop resp.)
 Execution and outcome of
the memory profiler (valgrind)
 Pages visited with the browser
(Firefox)
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

11. Experiment
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

12. References
 A. Pardo, C. Delgado Kloos:
“CCOLAB UC3M: Events recorded in an undergraduate
collaborative C Programming Course”.
In: DataTel 2010
 V.A. Romero, A. Pardo, D. Burgos, C. Delgado Kloos:
“Monitoring Student Progress Using Virtual Appliances:
A Case Study”.
Computers & Education 58: 4, May 2012, 1058–1067
 Lead researcher: Abelardo Pardo
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

13. 2. Visualization
 LearnGlass: Platform to facilitate the creation
of visualizations of learning events
 Implements generic requirements:
 User and permission management
 Data filters
 Visualization dashboard
Visualization
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

14. 14
LearnGlass Dashboard
3 variations
of the same
visualization
cdk@it.uc3m.es
Berlin, Germany EDUCON, Berlin, Germany, 2013-03-12--15
EDUCON, 2013-03-12--15

15. 15
Modular Architecture
 LearnGlass can be extended
through the installation of modules
 Each module provides:
 Visualization interface
 An optional light visualization for the dashboard
 There are currently two modules available:
 Activity report
 Learners with most and least events
cdk@it.uc3m.es
Berlin, Germany EDUCON, Berlin, Germany, 2013-03-12--15
EDUCON, 2013-03-12--15

16. 16
View of a module interface
cdk@it.uc3m.es
Berlin, Germany EDUCON, Berlin, Germany, 2013-03-12--15
EDUCON, 2013-03-12--15

17. References
 D. Leony, A. Pardo, L. de la Fuente Valentín,
D. Sánchez de Castro, C. Delgado Kloos:
“GLASS: A Learning Analytics Visualization Tool”.
In: International Conference on Learning Analytics and
Knowledge. ACM New York, USA, 2012, 162-163
 Lead researchers: Abelardo Pardo, Derick Leony
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

18. 3. Abstracting Out Data
 Genghis project at UC3M
 Khan Academy platform
deployed locally Aug 2012
 Flipping the classroom in
remedial course in Physics
 Analyse online student behaviour
online
Visualization Capture
of data
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

19. Low-level analytics provided
 Progress summary
 Daily activity report
 Skill progress over time
 Activity by day
 Badges earned
 Activity on exercises
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

20. Higher-level analytics inferred
 Time distribution
 Correct progress on the platform
 Efficiency in learning
 Gamification habits (influence of badges)
 Exercise solving habits (hint abuse, hint avoidance, …)
 Activity habits (explorer or recommendation follower)
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

21. Example: Efficiency in learning
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

22. References
 P.J. Muñoz-Merino, J.A. Ruipérez, C. Delgado Kloos:
“Inferring Higher Level Learning Information from
Low Level Data for the Khan Academy Platform”.
In: LAK2013: 3rd Conf. Learning Analytics and
Knowledge, 8-12 Apr. 2013, Leuven (Belgium)
 Lead researcher: Pedro Muñoz-Merino
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

23. 4. Just-in-Time Teaching
 ClassON system (in-CLass Analytics
for aSSessment and OrchestratioN)
 Use of learning analytics in face-to-face sessions
 awareness of students activity
 information to orchestrate in a better way
face-to-face
Visualization Capture
of data
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

24. Student component
 Web-based
 Integrated in the problem assignment
 Monitor student interactions
 progression
 questions
 timing
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

25. Teacher component
 Web app for tablets or smartphones
 Information attached to the physical space
 Teacher is aware of student’s
 location
 name and photo
 Progression
 questions
 waiting times
 And (s)he’s more informed to make decisions!
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

26. Teacher component interface
BUSY
2 3
5 2
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

27. References
 I. Gutiérrez Rojas, R.M. Crespo, C. Delgado Kloos:
“Orchestration and feedback in lab sessions:
improvements in quick feedback provision”.
In: EC-TEL 2011: 6th European Conf. of Technology
Enhanced Learning, 20-23 Sep. 2011, Palermo (Italy),
Proc. LNCS 6964. Springer 2011
 Lead researcher: Israel Gutiérrez
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

28. 5. Gamification
 ISCARE system
 Analytics for motivation and joy: Adaptation of the Swiss-
system of competition (used eg. in chess) to education
 Motivating and fair system:
 Different rounds. No elimination
 For each round, pairings of 2 people that compete
 Assign each person an opponent with close scoring
 Active learning: solving exercises
 Evaluation
 Interoperability: IMS QTI
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

29. ISCARE System
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

30. Analytics in ISCARE
 Evolution of scores for each student in each round
 The difficulty of the opponents
the student competed against
 Exercises assigned to each student and
if (s)he answered them correctly or not
 Information about each exercise:
number of times it was presented or solved correctly
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

31. Analytics in ISCARE
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

32. References
 M. Fernández Molina, P.J. Muñoz-Merino,
M. Muñoz-Organero, C. Delgado Kloos:
“Educational Justifications for the Design of the ISCARE
Computer Based Competition Assessment Tool”.
In: Internat. Conf. Web Based Learning, 2011, 289-294
 P.J. Muñoz-Merino, M. Fernández Molina,
M. Muñoz-Organero, C. Delgado Kloos:
“An Adaptive and Innovative Question-driven Competition-
based Intelligent Tutoring System for Learning”.
Expert Systems with Applications, 39(8), 2012, 6932-6948
 Lead researcher: Pedro Muñoz-Merino
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15

33. Conclusion
 In all important processes, the behaviour is monitored
(Quality control)
 Assessment is not enough
 Prerequisite for scaling: MOOCs
 Definitely helps in improving education
 Privacy concerns have to be taken care of
cdk@it.uc3m.es EDUCON, Berlin, Germany, 2013-03-12--15