Maka Eradze, Kairit Tammets
University of Naples Federico II/Tallinn University
Data Science and Social Research conference, Naples, Italy, 19.02.2015

DEFINITIONS
• Learning analytics is the measurement, collection,
analysis and reporting of data about learners and
their contexts, for purposes of understanding and
optimizing learning and the environments in which it
occurs.
• LA as a technology-enhanced learning (TEL)
research area that focuses on the development of
methods for analyzing and detecting patterns within
data collected from educational settings, and
leverages those methods to support the learning
experience. (Chatti et al 2015)

LA, ITS CHALLENGES AND
FUTURE DIRECTIONS
Learning analytics is inherently
intidisciplinary
• Operates on the verge of many different
fields
Still in its infancy
• It still has to establish itself as field with its
own traditions and theoretical frameworks

• Two systematic literature reviews found:
• Nistor et al(2015): The main issue with LA research is
the frequent lack of an explicit theoretical framework
from educational perspective. The authors call on the
educational and psychological theories for significant
progress of upcoming LA research
• Gasevic et al(2015): researchers from educational
sciences tend to publish more mature research and
show more rigor
• Fortunately, the field is evolving quickly and one measure
of this growth is the increasing breadth of methodologies
being applied to learning‐related data sources (Pardo et al,
2014)

• Big learning analytics
that come from
heterogeneous sources
• open learning
environments (MOOCs),
with data coming from
outside of platform.
• Mobile learning analytics
• Context modeling
• Privacy-aware analytics.
• Personalized learning analytics.
• Life-long learner modeling
• Learning analytics for open
assessment.
• Embedded Learning analytics
• Learning analytics design
patterns.
• Learning analytics evaluation.
Chatti, M. A., Lukarov, V., Thüs, H., Muslim, A., Yousef, A. M. F., Wahid, U., ... &
Schroeder, U. (2014). Learning Analytics: Challenges and Future Research Directions.

FACTORS
Different factors are involved in LA
Greller and Draschler

From Chatti

MOOCS AND LA
• Big data makes it possible to scale up learning
• MOOCs make it possible to open up learning
• “Teaching crowds”* needs new pedagogical models and new
ways to support the process
• Learning analytics can make it possible to help design, teach
and analyse the processes of “teaching crowds”
* Dron and Anderson

LEARNING ANALYTICS IN MOOCS:
CHALLENGES AND OPPORTUNITIES
Challenge of Effectiveness
● According to Clow (Clow 2013)* most of the MOOC analytics develop
around the formal education context and questions surrounding
predictive modeling, that is problematic in MOOCs.
● Chatti and Clow agree that LA analytics for MOOCs for now is rather
limited
● A focus on the perspectives of learners in MOOCs has the potential to
extend LA beyond completion rates and interventions to support
personalization, feedback, assessment, recommendation, awareness,
and self-reflection
● Facilitate individual decision making process
*Clow, D. (2013, April). MOOCs and the funnel of participation. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 185-
189). ACM.

LEARNING ANALYTICS IN MOOCS:
CHALLENGES AND OPPORTUNITIES
Business Models – opportunities
● Another and less well publicized MOOC business model
is dependent on “big data” or data analytics that can
provide indirect financial advantage. *
● Also, the utilisation of data analytics can improve the
accuracy of decision making, for instance by identifying
markets in which to target specific advertising campaigns*
Burd, E. L., Smith, S. P., & Reisman, S. (2015). Exploring business models for MOOCs in higher education. Innovative Higher Education, 40(1), 37-49.
*Loukis, E., Pazalos, K., & Salagara, A. (2012). Transforming e-services evaluation data into business analytics using value models. Electronic
Commerce Research and Applications, 11(2), 129-141.

LEARNING ANALYTICS IN MOOCS:
CHALLENGES AND
OPPORTUNITIES
Technology and pedagogy – challenges and opportunities
(Learning Analytics in Open Learning Environments (e.g. MOOCs)
● Incredible amounts of data for retrospective analysis
● Much of this data goes beyond platforms
● xMOOCs are seen as centralized courses, with predetermined paths
and closed systems - analytics would differ for these platforms
● cMOOCs are open ended, distributed and go beyond the platforms
● Learning analytics that focuses on learners, can provide them much
needed personalized feedback

EMMA is
• EU project (30 month CiP)
• It is a platform and an authoring system (LMS +CMS) to provide
hosting space for universities and cultural institutions to experiment
with MOOCs + (aggregation)
• EMMA integrates existing technological elements to create a pan-
European platform that offers opportunities for multicultural and
multilingual learning:
• UNINA Federica platform; UPV ASR + MT systems; CSP tracking;
TLU methodology + dashboards; IPSOS surveys
• Use learning analytics to create a feedback loop (for learners,
providers and developers)

EMMA LEARNING ANALYTICS
FRAMEWORK
LA in EMMA focuses:
1. On real time analytics through
learning analytics dashboards
for instructors and students –
feedback loop
2. Retrospective analytics of the
digital traces in EMMA platform
– evidence based learning
design
3. Is built on the Greller and
Draschler framework that LA
has a great potential as a
powerful tool to inform learners
and educators
From Greller and Draschler

XAPI
Event-driven analytics
<Actor (learner)> <verb> <object>, with <result>, in <context>

EMMA LA AND XAPI
● The Experience API is a service that allows for statements of
experience (typically learning experiences, but could be any
experience) to be delivered to and stored securely in a Learning
Record Store
● The Experience API is dependent on Learning Activity Providers
to create and track learning.
● Learning Activity Provider is a software object that
communicates with the LRS to record information about the
learning experience.
● Learning activity is a unit of instruction, experience or
performance that has to be tracked.
● A Statement consists of <Actor (learner)> <verb> <object>, with
<result>, in <context> to track an aspect of a learning
experience.

EMMA LA AND XAPI
● It repeats the structure of the sentence in practically any
language and is human readable
• Actor data is a unique information that describes a
specific subject, such as a student or group of students.
• Verb data classifies the type of activity the actor
participated in and often links to a human readable
description of the event.
• Object data will link to an artifact that is typically a
byproduct of or related to the activity

EMMA LA AND XAPI
● It is much more, than just a simple sentence-like structure. The
context is very important and it makes possible to relate one
activity to another*
• Quoting Verbert, Suthers and Rosen, Kevan and Ryan believe
that the issue of variety of educational data taxonomies and also
the distributed learning events collection challenges can be solved
by xAPI*
● Some researchers believe that it is aligned with constructivist
approach and activity theory
● xAPI has a lot of potential for The xAPI offers a renewed
opportunity to research, develop, and explore theories involving
learning beyond academia’s digital environments.
Kevan, J. M., & Ryan, P. R. Experience API: Flexible, Decentralized and Activity-Centric Data Collection.Technology,
Knowledge and Learning, 1-7.

EMMA learning analytics technical description
built-in tracking system that collects the data about users’ interactions
● The set of users’ interactions making up learning experience statements in
xAPI format are sent to Learning Record Store (LRS), which is used for
storing learning experiences of EMMA courses
● Some of the data (e.g. the structure of the MOOC and start and end date of
the course) to be visualized by the dashboard was retrieved from
● EMMA database with the support of web service
● implementation of visualizations
is mainly based on Highcharts
charts’ framework
● Social network analysis graph is
developed by using Sigma.js

Dashboards: Facilitator support:
● Overview of the course activities against the lessons of the MOOC
● Overview of the different interactions under the units and lessons
● Enrollments and unenrollments
● Activity streams
● Social network analysis

Dashboards: Learner support
● Progress compared with different lessons - how many units under each
lesson have been accessed by the learner, assignments submitted with
what result and participated
● Enrollments
● Activity stream of the recent activities – 100 latest activities of the MOOC
participants will be visualized as stream;
● Social network analysis based on comments and responses in
conversation module of the MOOC
● Overview of the popular resources and suggestions to access materials
that other participants have accessed, but this certain learner has not yet.

Clustering the participants of MOOCs
In the first phase, participants were clustered as: enrolled, observers
and contributors. As in the second phase, there were a bit more
interactions, the following clustering scheme was used:
• Enrolled – participant entered the MOOC up to five times;
• Observer – participant entered the MOOC more than five times, but
did not interact with the content or other participants;
• Contributor – participant contributed with the assignment, comment
or post to the MOOC at least once;
• Active – participant contributed with the assignment, comment or
post to the MOOC more than once
• Note: This sample does not include one course that had the greatest
success and highest enrollment rates: Coding in your classroom,
NOW!

MOOC Enrolled Observer Contributor Active
Computer support inquiry 29,3% 29,3% 40% 1,4%
Business Intelligence 45,6% 22,2% 10% 22,2%
Developing Blended Learning 44% 41,4% 8,9% 5,7%
E-learning 68,2% 27,4% 3,2% 1,2%
Lisbon and the Sea: a Story of
Arrivals and Departures,
42,4% 28% 20,3% 9,3%
The Organization of Cultural
Enterprises
66% 10,7% 10,7% 12,6%
General and Social Pedagogy 58,8% 27,5% 7,8% 5,9%
Social Innovation and cultural
Heritage
55,5% 28,7% 9,9% 5,9%
Mobile devices in your
everyday life
26,4% 59,5% 9% 5,1%
Climate changes: The Context
of Life Experience
46,9% 22,2% 16% 14,9%
Open Wine University 36,4% 12,4% 21,4% 29,8%
E-Portfolio self -development
study
56,4% 34,9% 6,4% 2,3%
Excel 62,8% 16,6% 7,5% 13,1%
Search on the internet 62,5% 14,1% 9,3% 14,1%

Some conclusions
• This indicates that the majority of Mooc participants just enrolled in
the courses or observed the content.
• One MOOC seems like an outlier with over 50% participants who
contributed to the course.
• TLU course Computer-Supported Inquiry had more than 40% of
participants who interacted with the platform
• Reasons
• technical – some participants did not find the functionalities or the
course assumed using blogs for interactions which is not the part of
learning analytics
• Selected pedagogical design – no interaction require

Social network analysis
is based on the statements related with commenting responding
and replying.
Example 1
Based on the figure, it is possible to
assume that
1. the course had more than one
facilitator and that some of the
participants were really active
commentators.
2. the direction of the
communication is not only from
learner to facilitator, but
participants interact with each
other quite a lot.

Example 2
We can see that
1. the course pedagogical
design affects the interactions
within the course. In that course
the students were supposed to
publish some of their tasks with
the conversation tool, so they
responded to the facilitator’s
request with their contribution.
2. It seems that participants did
not perceive others’
contributions relevant enough,
so they did not comment or
answer to nearly no comment.
each other quite a lot.

Interactions with content
The aim of the current analysis was to find out how participants
engage with the content during the course.
indicates that the
amount of interactions
and minutes spent on
content were slowly
decreasing after each
lesson.

• illustrates how in every second lesson
there was an increasing trend of
interactions and also then people spent
more time on the content.
• Course design may indicate that the
participants were supposed to do
something more or extra within course
settings, also it may

CONCLUSIONS
• Pedagogical neutrality - based on data that was analysed,
it can be said that EMMA platform supports different
pedagogical designs of MOOC.
• MOOC could be xMOOC, which focuses on content
consuming and could be also a cMOOC where
participants actively communicate and construct new
knowledge together.

FUTURE WORK
• Deeper insights into learning processes and uptake of
knowledge -
• Learning analytics dashboards evaluation finding out in which
way it supports the learning experience in EMMA platform and
also how MOOC facilitators plan their pedagogical
interventions of MOOCs to next iterations of the course: how
do they make sense of the data and how it is integrated to
course design process.
• Learning analytics data to be combined with the data of the
participants surveys in the platform: what expectations do
they have when they enter to the MOOC and what is their
learning path during the course

FUTURE WORK FROM WIDER
PERSPECTIVE
• The development of the EMMA learning analytics research,
framework and implementation also depends on the developments
in the field theory and shared experiences from different projects
using similar approaches.
• EMMA LA data translates into policy indicators for a common
European MOOC model
• The issues of data-standardization, data collection and analysis
that goes beyond the platform, specification and use of particular
verbs are the issues that also influence the EMMA platform LA
developments.

Contact:
maka@tlu.ee