2015 11-17 Venia Legendi Kairit Tammets

1. Learning Analytics for
Practice- and Policy-oriented
Educational Research
Venia legendi
Kairit Tammets
Candidate for the post of senior researcher
17th November 2015

2. Introduction
Rapid advances in technology provide us
necessary infrastructure to accept the online
learning and support the delivery of education at a
large scale;
The adoption of educational technologies enable
us new opportunities to gain insight into learning

3. Outline
̶ Learning analytics for practice:
̶ Competence-based learning in individual and
organizational level;
̶ Teacher’s learning and knowledge building practices
in socio-technical system;
̶ MOOCs and dashboards
̶ Learning analytics for policy-oriented
educational research

4. Competence-based
approach in education
• Evaluation of teacher’s professionalism is
formal and rigid - need for evidence-based
evaluation
• Teacher’s evaluation could be promoted by
taking on-the-job activities of teachers into
account (Tammets et al, 2011)
• Learning activities mapped with competencies
(Ley, Tammets, Lindstaedt, 2014; Tammets et
al, 2011)

5. Competence-based
approach in education
• Profiling individual competencies
• Digimina
• eDidaktikum
• IntelLEO (Intelligent Learning Extended
Organization)
• Profiling organizational competencies
• EpoAbi
• eDidaktikum
• IntelLEO

6. Digimina
• Web-based environment for
evaluating teachers’ digital
competencies
• Three evaluation types:
• Self-analysis (with the evidence)
• Peer-assessment
• Automatic assignment
• Based on the evaluation types,
individual competency profile will
be created
Põldoja, H., Väljataga, T., Laanpere, M., & Tammets, K. (2014). Web-based Self- and Peer-Assessment of
Teachers’ Digital Competencies. World Wide Web, 17(2), 255 - 269

7. Learning Path Creator
Individual learning paths:
• Define the required competences
• Identify gaps in your competence
development
• Plan your learning paths
• Monitor your development
System:
• Accumulates paths as
organizational paths
• Recommends appropriate learning
paths
Tammets, K., Pata, K., Laanpere, M., Tomberg V., Gaševic, D., & Siadaty, M. (2011). Designing the
Competence-driven Teacher Accreditation. H. Leung, E. Popescu, Y. Cao, R.W.H. Lau & W. Nejdl (Toim.).
Advances in Web-based Learning (132 - 141). Springer Verlag

8. EpoAbi
Bottom-up approach to course
design;
Course designers plan their
learning activities (nine different
activities), assessment methods,
technologies;
Course activities and assessment
methods are mapped and
associated with outcomes;
Systems store the course designs
and provides recommendations
for similar courses.
Tammets, K., & Pata, K. (2013). The trends and problems of planning outcome-based courses in elearning. Saar, E., Mõttus, R., &
Bern, E. (Toim.). Higher Education at a Crossroad: the Case of Estonia (281 - 301). Peter Lang Publishers House

9. eDidaktikum
Continuous and systemic
documentation of knowledge
and practices thorough formal
studies in teacher education;
Mapping learning activities and
digital resources with
competencies
Competency profile of pre-service
teacher will be created
Why not to create competency
profile of the course?
Curriculum?
Tammets, K., Tammets, P., & Laanpere, M. (2014). eDidaktikum – online community for
scaffolding pre-service teachers into digital culture. In: The proceedings of the ECER
conference: 01.- 05. September 2014, Porto

10. ̶ Individual’s documented knowledge and practices
enhances the organizational-level knowledge:
̶ Competency profiles: strengths of the staff; gaps in
competencies,
̶ Learning paths for acquiring certain competencies;
̶ Curriculum development

11. Learning and knowledge
building practices
• Individual internal learning and collaborative
knowledge building practices in the socio-technical
system
• Such practices support professional development, on
the job activities needed for evaluation and promote
the development of organizational knowledge
• For promoting teacher’s professional development,
systemic model is needed:
• Knowledge conversion SECI model (Nonaka & Takeuchi,
1995)

12. Learning and knowledge
building practices
Tammets, Pata & Laanpere, 2012

13. Socio-technical system
̶ Socio-technical system promotes enables to:
̶ Write reflections and share them within the community;
̶ Find and reuse those reflections with the aim to learn
from them, and for creating community knowledge;
̶ Participate in collaborative and knowledge building
activities related with knowledge, competences, or
actions with the community members;
̶ Plan professional development based on shared social
and community norms.

14. eDidaktikum
̶ Initially planned to be
learning resource repository
for teacher education
̶ Based on participatory
design approach, it was
designed as community-
based learning
environment of teacher
education;
̶ Competence-based learning

15. LKB in socio-technical
system
• Accumulated knowledge in the socio-
technical system acts as a scaffold for
teachers’ professional development
• Data stored in the system can be used for
supporting teachers’ learning and knowledge
building practices
Tammets, K., Pata, K., Laanpere, M. (2013). Promoting Teachers’ Learning and
Knowledge-building in the Socio-technical System. The International Review of Research
in Open and Distance Learning, 14(3), 251 - 272.

16. Learning analytics for policy-oriented research

17. Learning Analytics – what?
̶ Technologies and systems around us capture
learners’ interactions and their online activities
̶ Mining and analyzing log data of these systems
for identifying patterns, enables insights to into
educational practice
̶ Is: “Measurement, collection, analysis and
reporting of data about learners and their
contexts

18. Learning Analytics – Why?
̶ for purposes of understanding and optimizing
learning and the environments in which it occurs”
Siemens, G., & Gašević, D. (2012). Special Issue on Learning and Knowledge
Analytics. Educational Technology & Society, 15(3), 1–163.

19. Learning Analytics
The use of analytics in education has grown in
recent years for four primary reasons:
• a substantial increase in data quantity;
• improved data formats;
• advances in computing;
• increased sophistication of tools available for
analytics.
Baker, R., Siemens, G. (2014). Educational data mining and learning analytics. In Sawyer, K. (Ed.)
Cambridge Handbook of the Learning Sciences: 2nd Edition, pp. 253-274.

20. Existing research work in LA
̶ Predicting learner performance and modeling
learners – aim is to estimate the unknown value of a
variable that describes the learners, such as
performance, knowledge, scores or grades;
̶ Suggesting relevant learning resources –
recommender systems that analyze learner data to
suggest relevant learning resources or - paths
̶ Increasing reflection and awareness – analysis
and visualization of learning indicators to foster
awareness and reflection about learning processes
(resource access, time spending, knowledge level
indicators)

21. Existing research work in LA
̶ Enhancing social learning environments – make
people aware of their social context and enable then
to explore this context
̶ Detecting undesirable learner behaviors –
discover learners who have unusual behavior such
as misuse, cheating, dropping out or academic
failure
̶ Detecting affects of learners – boredom,
confusion, flow/engagement for adjusting
pedagogical strategies
Verbert, K., Manouselis, N., Drachsler, H., & Duval, E. (2012). Dataset-Driven
Research to Support Learning and Knowledge Analytics. Educational Technology &
Society, 15 (3), 133–148.

22. EMMA
(European
Mul.ple
MOOC
Pla2orm)
aims:
§ Create a pan-European platform to support ICT-
based innovation in higher education
§ Offer MOOCs provided by accredited
institutions from around Europe with different
teaching methodologies and learning design
approaches
§ Offer an extensive and multilingual
transcription/translation system
§ Offer an extensive monitoring system
§ Give opportunities to small universities and
cultural institutions to get in the MOOC market

23. EMMA LA methodology
• What are the research questions?
• What data is needed for answering research
questions?
• How this data can be collected?
• How to meaningfully present data?
• How to design feedback loop?
• How to learn from the process?

25. Dashboards
̶ Visualizations (dashboards) are used for visualizing
learning analytics results:
̶ Progress and overview of the course activities
̶ Performance
̶ Social structures
̶ Recommendations
̶ BUT:
̶ Design and use of learning analytics dashboards is far
less understood;
̶ What changes after using dashboards in teaching and
learning?

27. Issues around large scale LA
- Numerous small-scale R&D projects have
demonstrated successful outcomes, but are
dependent on contextual factors -> no strong
evidence of the overall effectiveness of LA at scale
̶ Few universities have made use of LA at scale;
̶ No detailed implementation accounts available;
̶ Organizational leaders must require the vision to
see how small-scale projects might be scaled to
improve teaching and learning across an institution
Ferguson, Rebecca; Macfadyen, Leah P.; Clow, Doug; Tynan, Belinda; Alexander, Shirley
and Dawson, Shane (2015). Setting learning analytics in context: overcoming the barriers
to large-scale adoption. Journal of Learning Analytics, 1(3) pp. 120–144.

28. LA in policy-oriented
educational research
̶ LA as a powerful tool for educational management
̶ Merging educational research and practice with LA data
provides novel and real-time approaches to assess issues
impacting eudcation: retentions, 21st cent skills,
personalised learning;
̶ Automating measurements and predictions is promising,
but focus is on outcomes; take into account learning and
teaching process

29. LA in policy-oriented
educational research
̶ Pedagogical approach: see the pedagogy behind the
numbers;
̶ Field of learning analytics needs to ground data
collection, measurement, analysis, reporting and
interpretation processes within the existing research on
learning
̶ Take the numbers into account in policy-level decisions
(curriculum development)
̶ Learning analytics is question-driven, not data or
technology driven

30. The Rapid Outcome
Mapping Approach (ROMA)
̶ Model for guiding an iterative approach to
planning the systemic institutional
implementation of learning analytics;
̶ Seven‐step model is focused on evidence‐
based policy change
̶ Goal is moving to broader institutional
implementation

32. In agenda
̶ What happens after dashboards:
̶ Interpretation skills needed for using dashboards?
̶ In which way instructional designs change as a result of using
dashboard?
̶ How dashboard may influence learners’ decision-making skills, learning
from failure and becoming self-sufficient?
̶ Integration of LA with educational research: e.g. how students’
learn in online settings and how do they perceive their learning;
̶ Integration of LA to higher education;
̶ Bringing LA to policy level: LA results of different learning
environments as part of curriculum development process in DTI

33. Conclusion
̶ ROMA model – systemic implementation of LA in
institutional level;
̶ Participatory design of LA tools and practices –
learners, teachers, instructional designers, technologists,
researchers and policy-decision makers need to work
together for avoiding undesirable practices of LA;
̶ Integration with the educational research;
̶ When our focus is on improving learning, the critical
results we need to monitor and measure are the results
that reflect positive educational change
̶ .

34. References
̶ Tammets, K., Pata, K., Laanpere, M., Tomberg V., Gaševic, D., & Siadaty, M.
(2011). Designing the Competence-driven Teacher Accreditation. H. Leung, E.
Popescu, Y. Cao, R.W.H. Lau & W. Nejdl (Toim.). Advances in Web-based Learning
(132 - 141). Springer Verlag
̶ Ley, T., Tammets, K., & Lindstaedt, S. (2014). Orchestrating collaboration and
community technologies for individual and organisational learning. LittleJohn, A.,
& Margaryan A. (Toim.). Technology-Enhanced Professional Learning: Processes,
Practices, and Tools (117 - 131). New York: Routledge

35. Thank you!
Kairit Tammets
kairit@tlu.ee