The emergence of learning analytics afforded for the analysis of digital traces of user interaction with technology. This analysis offers many opportunities to advance understanding and enhance learning and the environments in which learning occurs. Existing research has shown how learning analytics can provide contributions to different areas of education such as prediction of student success, uncovering learning strategies, understanding affective states, and unpacking the role social networks in learning. While these results have shown much promise, one critical challenge remains unclear – how learning analytics can help track learning progression and inform assessment especially from the perspective of the 21st century skills. This talk will explore opportunities and challenges for the integration of methods commonly used in learning analytics to analyze different digital traces with methods commonly used in assessment. The talk particularly focuses on open learning environments where analytics-based assessment is rather underexplored in contrast to assessment in specialized (intelligent tutoring) systems where the combined use of data mining and assessment has been established for some time now.

1. Can learning analytics offer
meaningful assessment?
Dragan Gašević
@dgasevic
EARLI SIG Assessment
August 29, 2018
Helsinki, Finland

2. Feedback loops between
students and instructors
are missing/weak!

3. Learning
environment
Educators
Learners
Student
Information
Systems

4. Blogs
Videos/slides
Mobile
Search
Educators
Learners
Networks
Student
Information
Systems
Learning
environment

5. Blogs
Mobile
Search
Networks
Educators
Learners
Student
Information
Systems
Learning
environment
Videos/slides

6. Data in education not new, but…
Real-time insights and longitudinal nature

7. Analytics for
custom made environments
e.g., intelligent tutoring systems

8. Moment by moment learning
Baker , R. S., Hershkovitz, A., Rossi, L. M., Goldstein, A. B, & Gowda, S. M. (2013) Predicting Robust Learning With the Visual Form of the Moment by-
Moment Learning Curve, Journal of the Learning Sciences, 22(4), 639-666, DOI: 10.1080/10508406.2013.836653

9. Open ended and general purpose
environments
Different from custom made environments

10. Learning analytics purpose
Understanding and supporting learning

11. Current state in learning analytics
Moving away from deficit models

12. High interest in adoption of
learning analytics

13. Is everything that shiny?

14. CHALLENGES IN
LEARNING ANALYTICS

15. Scrutinize learning analytics against
assessment properties

16. Challenges
Validity – Progression

17. Challenges
Validity – Progression

18. Framework
Unified theory of construct validity
Messick, S. (1994). Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into
Score Meaning. ETS Research Report Series, 1994(2), i-28. https://doi.org/10.1002/j.2333-8504.1994.tb01618.x

19. Content
Engagement is primarily discussed in
learning analytics

20. Content
But, how is engagement
defined and measured?

21. Content

22. Content
Are simple counts of clicks
measures of engagement?

23. Content
Who makes decisions about
instrumentation?

24. Purposeful measurement
Can we trust some of measures?
Time on task
Kovanovic, V., Gašević, D., Dawson, S., Joksimovic, S., & Baker, R. S. (2016). Does Time-on-task Estimation Matter? Implications on Validity of Learning
Analytics Findings. Journal of Learning Analytics, 2(3), 81–110. https://doi.org/10.18608/jla.2015.23.6

25. Purposeful measurement
Can we trust some of measures?
Network centrality
Fincham, E., Gašević, D., Pardo, A. (2018). From Social Ties to Network Processes: Do Tie Definitions Matter?. Journal of Learning Analytics, 5(2), 9–28.

26. Consequential (aka actionable)
Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions
in predicting learning success. The Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002
How can we act based on
the count of logins?

27. Consequential
Tanes, Z., Arnold, K. E., King, A. S., & Remnet, M. A. (2011). Using Signals for appropriate feedback: Perceptions and practices. Computers &
Education, 57(4), 2414-2422.
Can teaching be improved?

28. Generalizability
Difficulty to replicate findings

29. Open Academic Analytics Initiative
http://nextgenlearning.org/grantee/marist-college

30. Inconsistent associations of
network centrality on performance
External validity
Joksimović, S., Manataki, A., Gašević, D., Dawson, S., Kovanović, V., & De Kereki, I. F. (2016). Translating network position into performance:
importance of centrality in different network configurations. In Proceedings of the Sixth International Conference on Learning Analytics &
Knowledge (pp. 314-323). ACM.

31. Structural validity
Do existing* measures correspond to
trace-based measures?
*mostly self-reported
Gašević, D., Jovanović, J., Pardo, A., & Dawson, S. (2017). Detecting Learning Strategies with Analytics: Links with Self-reported Measures and Academic
Performance. Journal of Learning Analytics, 4(2), 113–128.

32. Understanding learning strategies
Detection of learning tactics Detection of learning strategy
Fincham, E., Gašević, D., Jovanović, Pardo, A. (2018). From Study Tactics to Learning Strategies: An Analytical Method for Extracting Interpretable
Representations. IEEE Transactions on Learning Technologies (in press), https://doi.org/10.1109/TLT.2018.2823317

33. Challenges
Validity – Progression

34. Challenge
How do we measure progression?

35. Challenge
Does increase in number of clicks
means progression?

36. Increase in activity ≠ increase in learning
Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on
cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89. https://doi.org/10.1016/j.iheduc.2015.06.002

37. Increase in activity ≠ increase in learning
Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on
cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89. https://doi.org/10.1016/j.iheduc.2015.06.002

38. Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet
and Higher Education, 33, 74–85.
Increase in activity ≠ increase in learning

39. Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet
and Higher Education, 33, 74–85.
Increase in activity ≠ increase in learning
A

40. Negative predictors of learning
Student-content (counts)
Student-teacher (time)
Joksimović, S., Gašević, D., Loughin, T. M., Kovanović, V., & Hatala, M. (2015). Learning at distance: Effects of interaction traces on academic achievement.
Computers & Education, 87, 204–217. https://doi.org/10.1016/j.compedu.2015.07.002

41. Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., Siemens, G. (2016). Towards Automated Content Analysis of Discussion
Transcripts: A Cognitive Presence Case. Proceedings of the 6th International Conference on Learning Analytics & Knowledge (pp. 15-24).
Cognitive presence

42. DIRECTIONS

43. Strengthening links between
learning analytics and assessment

44. Critical dimensions
Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the Learning Analytics Puzzle: A Consolidated Model of a Field of Research and Practice.
Learning: Research and Practice, 3(2), 63-78. doi:10.1080/23735082.2017.1286142

45. Challenges
Validity – Progression

46. Content
Extraction of
theoretically informed traces

47. Purposeful measurement
Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning
Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11

48. Purposeful measurement
Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning
Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11

49. Purposeful measurement
Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning
Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11

50. Purposeful measurement
Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning
Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11

51. Purposeful measurement
Transparency in measurement
Kovanovic, V., Gašević, D., Dawson, S., Joksimovic, S., & Baker, R. S. (2016). Does Time-on-task Estimation Matter? Implications on Validity of Learning
Analytics Findings. Journal of Learning Analytics, 2(3), 81–110. https://doi.org/10.18608/jla.2015.23.6

52. Gašević, D., Dawson, S., Rogers, T., Gašević, D. (2016). Learning analytics should not promote one size fits all: The effects of course-specific technology
use in predicting academic success. The Internet and Higher Education, 28, 68–84.
Generalizability
Instructional conditions shape
learning analytics results

53. Structural validity
Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–
419. doi:10.1016/j.learninstruc.2012.03.004
Achievement goal
orientation (2x2)

54. Quid pro quo

55. Traced measures and quid pro quo

56. External validity
Network centrality with weak ties
creates advantage only
Joksimović, S., Manataki, A., Gašević, D., Dawson, S., Kovanović, V., & De Kereki, I. F. (2016, April). Translating network position into performance:
importance of centrality in different network configurations. In Proceedings of the Sixth International Conference on Learning Analytics &
Knowledge (pp. 314-323). ACM.

57. Measurement of engagement
Joksimović, S., Poquet, O., Kovanović, V., Dowell, D., Mills, C., Gašević, D., Dawson, S., Graesser, A. C., Brooks , C. (2018). How do we measure learning
at scale? A systematic review of the literature. Review of Educational Research, 88(1), 43-86.

58. Challenges
Validity – Progression

59. Progression
Trace data based measures of
the crowd-sourced learning skill
E.g., Dreyfus model of skill acquisition
Milligan, S. (2015). Crowd-sourced learning in MOOCs: learning analytics meets measurement theory. In Proceedings of the 5th International Conference
on Learning Analytics And Knowledge (pp. 151-155). ACM.

60. Progression
Topic modeling to extract
Guttman scales from online discussions
He, J., Rubinstein, B. I., Bailey, J., Zhang, R., Milligan, S., & Chan, J. (2016). MOOCs Meet Measurement Theory: A Topic-Modelling Approach. Proceedings
of the 30th AAAI Conference on Artificial Intelligence (pp. 1195-1201).

61. Tracking progression
Ferreira, R., Kovanović, V., Gašević, D., & Rolim, V. (2018). Towards Combined Network and Text Analytics of Student Discourse in Online Discussions.
In Proceeding of the 19th International Conference on Artificial Intelligence in Education (pp. 111-126).

62. Tracking progression
Ferreira, R., Kovanović, V., Gašević, D., & Rolim, V. (2018). Towards Combined Network and Text Analytics of Student Discourse in Online Discussions.
In Proceeding of the 19th International Conference on Artificial Intelligence in Education (pp. 111-126).
Integration
Exploration
Triggering event

63. Tracking progression
Ferreira, R., Kovanović, V., Gašević, D., & Rolim, V. (2018). Towards Combined Network and Text Analytics of Student Discourse in Online Discussions.
In Proceeding of the 19th International Conference on Artificial Intelligence in Education (pp. 111-126).

64. Tracking progression

65. FINAL REMARKS

66. Opportunities afforded by
continuous streams of data

67. Data science methods can be helpful
but not sufficient
von Davier, A. A. (2016). Computational psychometrics in support of collaborative educational assessments. Journal of Educational Measurement,
54(1), 3-11.

68. Links with methods from assessment
and psychometric needed

69. Tolerance to
some measurement imperfections

70. Generalizability in machine learning is
an open research challenge too

72. Can learning analytics offer
meaningful assessment?
Dragan Gašević
@dgasevic
EARLI SIG Assessment
August 29, 2018
Helsinki, Finland