Download as PDF, PPTX
![Learning analytics is the measurement, collection, analysis
and reporting of data about learners and their contexts,
for purposes of understanding and optimising learning and
the environments in which it occurs” - George Siemens [1]
[1] G. Siemens. “Learning analytics: envisioning a research discipline and
a domain of practice”. Proceedings of the 2nd International Conference on
Learning Analytics and Knowledge . ACM. 2012, pp. 4–8.
DEFINITION
page 1 of the thesis text
2](https://image.slidesharecdn.com/publicdefense-151124120110-lva1-app6892/85/Public-PhD-defense-2-320.jpg)
![[3,4]
Time
spent
Artefacts
Produced
Social
Interaction
Resource
use
Exercise/
Test
results
Ch. 2
Ch. 3
Ch. 4
[3] K. Verbert, E. Duval, J. Klerkx, S. Govaerts, and J. L. Santos. “Learning Analytics Dashboard Applications”. In:
American Behavioral Scientist 57.10 (2013), pp. 1500–1509.
[4] K. Verbert, S. Govaerts, E. Duval, J. L. Santos, F. Van Assche, G. Parra, and J. Klerkx. “Learning dashboards: an
overview and future research opportunities”. In: Personal and Ubiquitous Computing 18.6 (2014), pp. 1499–1514.
Approach
page 15 of the thesis text
59](https://image.slidesharecdn.com/publicdefense-151124120110-lva1-app6892/85/Public-PhD-defense-59-320.jpg)
![[3,4]
Time
spent
Artefacts
Produced
Social
Interaction
Resource
use
Exercise/
Test
results
Ch. 2
Ch. 3
Social
activity
Ch. 4
Outcomes
page 16 of the thesis text
61](https://image.slidesharecdn.com/publicdefense-151124120110-lva1-app6892/85/Public-PhD-defense-61-320.jpg)
![“Results of our analyses [94, 95] report that
dashboards for teachers are designed to raise
awareness of the activities taking place in the course,
analyse activity and plan interventions, among others.
Related to activity analysis, we explored what
teachers can actually learn from visualisations.”
Problem66
rq2 problem statement - see page 17 of the thesis text](https://image.slidesharecdn.com/publicdefense-151124120110-lva1-app6892/85/Public-PhD-defense-66-320.jpg)
![Time
spent
Artefacts
Produced
Social
Interaction
Resource
use
Exercise/
Test
results
Drop-outs
[5,6,7]
Language use
[8]
Social
interaction [9]
[5] C. Alario-Hoyos et al. “Analysing the Impact of Built-In and External Social Tools in a MOOC on Educational Technologies”. In: ECTEL’13 . Vol. 8095. LNCS. Springer, 2013, pp. 5–18.
[6] D. Clow. “MOOCs and the funnel of participation”. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. LAK ’13. ACM, 2013, pp. 185–189.
[7] H. Spoelstra et al. “Team formation instruments to enhance learner interactions in open learning environments”. In: Computers in Human Behavior 45 (2015), pp. 11–20.
[8] P. Levy. “Technology-Supported Design for Inquiry-Based Learning”. In: Exploring Learning & Teaching in Higher Education . Springer, 2015, pp. 289–304.
[9] N. Michinov et al. “Procrastination, participation, and performance in online learning environments”. In: Computers & Education 56.1 (Jan. 2011), pp. 243–252.
67
Approach
table with data from page 17 and chapter 5 of the thesis text](https://image.slidesharecdn.com/publicdefense-151124120110-lva1-app6892/85/Public-PhD-defense-67-320.jpg)
![Time
spent
Artefacts
Produced
Social
Interaction
Resource
use
Exercise/
Test
results
Drop-outs
[5,6,7]
attention to the first units of the lessons
Language use
[8]
Social
interaction [9]
68
Outcome
table with data from page 18 of the thesis text](https://image.slidesharecdn.com/publicdefense-151124120110-lva1-app6892/85/Public-PhD-defense-68-320.jpg)
![Time
spent
Artefacts
Produced
Social
Interaction
Resource
use
Exercise/
Test
results
Drop-outs
[5,6,7]
attention to the first units of the lessons
Language use
[8]
? ?
Social
interaction [9]
69
Outcome
table with data from page 18 of the thesis text](https://image.slidesharecdn.com/publicdefense-151124120110-lva1-app6892/85/Public-PhD-defense-69-320.jpg)
![Time
spent
Artefacts
Produced
Social
Interaction
Resource
use
Exercise/
Test
results
Drop-outs
[5,6,7]
attention to the first units of the lessons
Language use
[8]
? ?
Social
interaction [9]
70
Outcome
table with data from page 18 of the thesis text
thresholds thresholds](https://image.slidesharecdn.com/publicdefense-151124120110-lva1-app6892/85/Public-PhD-defense-70-320.jpg)
![“Therefore, we consider relevant to evaluate how
students perceived automatic and manual trackers.”
Problem72
rq3 problem statement - see page 19 of the thesis text
Manual Automatic
PROS Privacy tracking fatigue [10]
CONTRAS tracking fatigue [10] Privacy
[10] E. K. Choe, N. B. Lee, B. Lee, W. Pratt, and J. A. Kientz. “Understanding quantified-selfers’ practices in collecting and exploring personal data”. In:
Proceedings of the 32nd annual ACM conference on Human factors in computing systems . ACM. 2014, pp. 1143–1152.](https://image.slidesharecdn.com/publicdefense-151124120110-lva1-app6892/85/Public-PhD-defense-72-320.jpg)
Uploaded byJose Luis Santos Odriozola
Public PhD defense
The document presents a comprehensive examination of learning analytics and learning dashboards from a Human-Computer Interaction (HCI) perspective, highlighting their potential to optimize learning environments. It addresses key research questions regarding the visualization of learning activities for both learners and teachers, as well as methodologies for tracking and managing data associated with these activities. The study is informed by various case studies and publications that demonstrate the applications and implications of learning dashboards in educational settings.
More Related Content
![[Extended] Bottom-up growth of learning analytics at two Australian universit...](https://cdn.slidesharecdn.com/ss_thumbnails/bottom-upgrowthofla-160416081432-thumbnail.jpg?width=640&height=640&fit=bounds)
More from Jose Luis Santos Odriozola
![[DSC Europe 25] Ekaterina Bubenko - Behind the Curtain: How Data Roles Collab...](https://cdn.slidesharecdn.com/ss_thumbnails/anmv6x8dstqbbzchoklr-ekaterina-bubenko-behind-the-curtain-how-data-roles-collaborate-in-the-ai-era-a-260123083019-4b252ec7-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Milos Belcevic - Product Professional's Journey to Full-Stack...](https://cdn.slidesharecdn.com/ss_thumbnails/1zovd6fgsycdg4wvgvls-milos-belcevic-product-professionals-journey-to-full-stack-product-developer-260123083019-d993120d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Josip Saban - Career building for data professionals.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/zroflcttkm1vmli0txea-josip-saban-career-building-for-data-professionals-260123083019-587cdb8c-thumbnail.jpg?width=640&height=640&fit=bounds)
Public PhD defense
- 1. Exploring Learning Analytics andLearning Dashboards from a HCI Perspective Jose Luis Santos PUBLIC PHD DEFENSE 1http://bit.do/santos_scholar http://www.slideshare.net/jlsantoso
- 2. Learning analytics isthe measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” - George Siemens [1] [1] G. Siemens. “Learning analytics: envisioning a research discipline and a domain of practice”. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge . ACM. 2012, pp. 4–8. DEFINITION page 1 of the thesis text 2
- 3.
- 4.
- 5.
- 6.
- 7.
- 8. Classrooms and learningcommunities SCOPE see background section (pg.3) of the thesis text 8
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15. Classrooms and learningcommunities Learning Dashboards as Personal Informatics Tools see background section (pg.3) of the thesis text 15 SCOPE
- 16. https://www.flickr.com/photos/pere/523019984 The Iceberg -Abelardo Pardo’s metaphorLMS The Open 16
- 17.
- 18.
- 19.
- 20.
- 21. Classrooms and learningcommunities Learning Dashboards as Personal Informatics Tools Open Learning Environments see background section (pg.3) of the thesis text 21 SCOPE
- 22. MOTIVATION ability motivation triggers fail here triggers succeed here BJFogg’s model http://www.behaviormodel.org/ 22
- 23. Classrooms and learningcommunities Learning Dashboards as Personal Informatics Tools Open Learning Environments Motivation see background section (pg.3) of the thesis text 23 SCOPE
- 24. EVALUATION INSTRUMENTS 24 • GoogleAnalytics • Own tracking systems
- 25. EVALUATION INSTRUMENTS 25 • GoogleAnalytics • Own tracking systems • SUS questionnaire
- 26. EVALUATION INSTRUMENTS 26 • GoogleAnalytics • Own tracking systems • SUS questionnaire • Customised questionnaires
- 27. EVALUATION INSTRUMENTS 27 • GoogleAnalytics • Own tracking systems • SUS questionnaire • Customised questionnaires • Interviews
- 28. METHODOLOGY Iterative Real environments No controlled variables leads toknowledge that can be used by practitioners Design-based research 28
- 29. METHODOLOGY Iterative Real environments No controlled variables leads toknowledge that can be used by practitioners Design-based research 29
- 30. METHODOLOGY Iterative Real environments No controlled variables leads toknowledge that can be used by practitioners Design-based research 30
- 31. METHODOLOGY Iterative Real environments No controlled variables leads toknowledge that can be used by practitioners Design-based research 31
- 32.
- 33. OPEN LEARNING -CASE STUDY 133
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41. 41 What did we learn? Whatcame next?
- 42. OPEN LEARNING -CASE STUDY 242
- 43. OPEN LEARNING -CASE STUDY 243
- 44. OPEN LEARNING -CASE STUDY 244
- 45.
- 46.
- 47. 47 What did we learn? Whatcame next?
- 48.
- 49. OPEN LEARNING -CASE STUDY 349
- 50.
- 51.
- 52.
- 53. EBL - EnquiryBased Learning53
- 54. RESEARCH QUESTIONS RQ1: Whatcharacteristics of learning activities can be visualised usefully for learners? RQ2: What characteristics of learning activities can be visualised usefully for teachers? RQ3: What are the affordances of and user problems with tracking data automatically and manually? RQ4: What are the key components of a simple and flexible architecture to collect, store and manage learning activity? 54
- 55. RESEARCH QUESTIONS RQ1: Whatcharacteristics of learning activities can be visualised usefully for learners? RQ2: What characteristics of learning activities can be visualised usefully for teachers? RQ3: What are the affordances of and user problems with tracking data automatically and manually? RQ4: What are the key components of a simple and flexible architecture to collect, store and manage learning activity? 55
- 56. Approach56 128 students3 learninganalytics dashboards 5 case studies real settingsusability perceived usefulness
- 57. Publications57 Santos et al.2012. “Goal-oriented visualizations of activity tracking: a case study with engineering students”, In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ’12). ACM, New York, NY, USA, 143-152. Santos et al. 2013a. “Addressing learner issues with StepUp!: an evaluation”, In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK ’13). ACM, New York, NY, USA, 14-22. Santos et al. 2013b. “Evaluating the use of open badges in an open learning environment” , In proceedings of of the Eight European Conference on Technology Enhanced Learning, Scaling up Learning for Sustained Impact, Springer Berlin Heidelberg, Berlin, Germany. 314-327. Santos et al. 2015 “Tracking Data in Open Learning Environments" Journal of Universal Computer Science, Vol. 21, No. 7, pp. 976-996 Santos et al. 2014. “Success, activity and drop-outs in MOOCs an exploratory study on the UNED COMA courses”. In Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (LAK’14). ACM, New York, NY, USA, 98-102
- 58. “As described insection 1.1, learning dashboards visualise learning traces, actions that students perform while they learn. In this context, RQ1 explores the usefulness of such traces in five different open learning courses.” Problem58 rq1 problem statement - see page 14 of the thesis text
- 59. [3,4] Time spent Artefacts Produced Social Interaction Resource use Exercise/ Test results Ch. 2 Ch. 3 Ch.4 [3] K. Verbert, E. Duval, J. Klerkx, S. Govaerts, and J. L. Santos. “Learning Analytics Dashboard Applications”. In: American Behavioral Scientist 57.10 (2013), pp. 1500–1509. [4] K. Verbert, S. Govaerts, E. Duval, J. L. Santos, F. Van Assche, G. Parra, and J. Klerkx. “Learning dashboards: an overview and future research opportunities”. In: Personal and Ubiquitous Computing 18.6 (2014), pp. 1499–1514. Approach page 15 of the thesis text 59
- 60. Time spent Artefacts Produced Social Interaction Resource use Exercise/ Test results Ch. 2 Ch. 3Comparison Ch. 4 Outcomes page 16 of the thesis text 60
- 61. [3,4] Time spent Artefacts Produced Social Interaction Resource use Exercise/ Test results Ch. 2 Ch. 3 Social activity Ch.4 Outcomes page 16 of the thesis text 61
- 62. Time spent Artefacts Produced Social Interaction Resource use Exercise/ Test results Ch. 2 Ch. 3individual vs group work Ch. 4 Outcomes page 16 of the thesis text 62
- 63. RESEARCH QUESTIONS RQ1: Whatcharacteristics of learning activities can be visualised usefully for learners? RQ2: What characteristics of learning activities can be visualised usefully for teachers? RQ3: What are the affordances of and user problems with tracking data automatically and manually? RQ4: What are the key components of a simple and flexible architecture to collect, store and manage learning activity? 63
- 64. Approach64 exploratory study 2 languagelearning MOOCs 56876 students enrolled
- 65. Publications65 Santos et al.2012. “Goal-oriented visualizations of activity tracking: a case study with engineering students”, In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ’12). ACM, New York, NY, USA, 143-152. Santos et al. 2013a. “Addressing learner issues with StepUp!: an evaluation”, In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK ’13). ACM, New York, NY, USA, 14-22. Santos et al. 2013b. “Evaluating the use of open badges in an open learning environment” , In proceedings of of the Eight European Conference on Technology Enhanced Learning, Scaling up Learning for Sustained Impact, Springer Berlin Heidelberg, Berlin, Germany. 314-327. Santos et al. 2014. “Success, activity and drop-outs in MOOCs an exploratory study on the UNED COMA courses”. In Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (LAK’14). ACM, New York, NY, USA, 98-102 Santos et al. 2015 “Tracking Data in Open Learning Environments" Journal of Universal Computer Science, Vol. 21, No. 7, pp. 976-996
- 66. “Results of ouranalyses [94, 95] report that dashboards for teachers are designed to raise awareness of the activities taking place in the course, analyse activity and plan interventions, among others. Related to activity analysis, we explored what teachers can actually learn from visualisations.” Problem66 rq2 problem statement - see page 17 of the thesis text
- 67. Time spent Artefacts Produced Social Interaction Resource use Exercise/ Test results Drop-outs [5,6,7] Language use [8] Social interaction [9] [5]C. Alario-Hoyos et al. “Analysing the Impact of Built-In and External Social Tools in a MOOC on Educational Technologies”. In: ECTEL’13 . Vol. 8095. LNCS. Springer, 2013, pp. 5–18. [6] D. Clow. “MOOCs and the funnel of participation”. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. LAK ’13. ACM, 2013, pp. 185–189. [7] H. Spoelstra et al. “Team formation instruments to enhance learner interactions in open learning environments”. In: Computers in Human Behavior 45 (2015), pp. 11–20. [8] P. Levy. “Technology-Supported Design for Inquiry-Based Learning”. In: Exploring Learning & Teaching in Higher Education . Springer, 2015, pp. 289–304. [9] N. Michinov et al. “Procrastination, participation, and performance in online learning environments”. In: Computers & Education 56.1 (Jan. 2011), pp. 243–252. 67 Approach table with data from page 17 and chapter 5 of the thesis text
- 68. Time spent Artefacts Produced Social Interaction Resource use Exercise/ Test results Drop-outs [5,6,7] attention to thefirst units of the lessons Language use [8] Social interaction [9] 68 Outcome table with data from page 18 of the thesis text
- 69. Time spent Artefacts Produced Social Interaction Resource use Exercise/ Test results Drop-outs [5,6,7] attention to thefirst units of the lessons Language use [8] ? ? Social interaction [9] 69 Outcome table with data from page 18 of the thesis text
- 70. Time spent Artefacts Produced Social Interaction Resource use Exercise/ Test results Drop-outs [5,6,7] attention to thefirst units of the lessons Language use [8] ? ? Social interaction [9] 70 Outcome table with data from page 18 of the thesis text thresholds thresholds
- 71. RESEARCH QUESTIONS RQ1: Whatcharacteristics of learning activities can be visualised usefully for learners? RQ2: What characteristics of learning activities can be visualised usefully for teachers? RQ3: What are the affordances of and user problems with tracking data automatically and manually? RQ4: What are the key components of a simple and flexible architecture to collect, store and manage learning activity? 71
- 72. “Therefore, we considerrelevant to evaluate how students perceived automatic and manual trackers.” Problem72 rq3 problem statement - see page 19 of the thesis text Manual Automatic PROS Privacy tracking fatigue [10] CONTRAS tracking fatigue [10] Privacy [10] E. K. Choe, N. B. Lee, B. Lee, W. Pratt, and J. A. Kientz. “Understanding quantified-selfers’ practices in collecting and exploring personal data”. In: Proceedings of the 32nd annual ACM conference on Human factors in computing systems . ACM. 2014, pp. 1143–1152.
- 73. 73 Approach table with datafrom page 20 of the thesis text Lab sessions Blended learning courses no learning activity outside of the classroom big part of the learning activity outside of the classroom automatic trackers manual trackers Rabbit Eclipse plug-in * * https://marketplace.eclipse.org/content/rabbit
- 74. Outcome Lab sessions Blendedlearning courses Approach Automatic trackers Manual trackers No privacy concerns No tracking fatigue lack of tracking Over reporting 74
- 75. RESEARCH QUESTIONS RQ1: Whatcharacteristics of learning activities can be visualised usefully for learners? RQ2: What characteristics of learning activities can be visualised usefully for teachers? RQ3: What are the affordances of and user problems with tracking data automatically and manually? RQ4: What are the key components of a simple and flexible architecture to collect, store and manage learning activity? 75
- 76. Publications76 Santos et al.2015 “Tracking Data in Open Learning Environments" Journal of Universal Computer Science, Vol. 21, No. 7, pp. 976-996 Santos et al. 2012. “Goal-oriented visualizations of activity tracking: a case study with engineering students”, In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ’12). ACM, New York, NY, USA, 143-152. Santos et al. 2013a. “Addressing learner issues with StepUp!: an evaluation”, In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK ’13). ACM, New York, NY, USA, 14-22. Santos et al. 2013b. “Evaluating the use of open badges in an open learning environment” , In proceedings of of the Eight European Conference on Technology Enhanced Learning, Scaling up Learning for Sustained Impact, Springer Berlin Heidelberg, Berlin, Germany. 314-327. Santos et al. 2014. “Success, activity and drop-outs in MOOCs an exploratory study on the UNED COMA courses”. In Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (LAK’14). ACM, New York, NY, USA, 98-102
- 77. 77 image at page21 of the thesis text Approach Experience report two environments
- 78. 78 Rabbit Eclipse plugin RescueTime Wordpress API BlogspotAPI Medium / RSS Twitter Toggl Trackers REST services dashboard badge system Internethosted in the cloud Google App Engine 1. Common data schema 2 3 three elements described in page 22 of the thesis text Outcome
- 79.
- 80. image at page137 of the thesis text 128 students actually used the learning dashboards 56876 students enrolled in the MOOC courses the architecture was deployed in more than 10 case studies 3 learning analytics dashboards 80
- 81. Publications and RQs C- Conference, J - Journal RQ1 Chapter 2: Santos et al. 2012 (C) Chapter 3: Santos et al. 2013a (C) Chapter 4: Santos et al. 2013b (C) RQ2 Chapter 5: Santos et al. 2014 (C) RQ3 Chapter 2: Santos et al. 2012 (C) Chapter 3: Santos et al. 2013a (C) RQ4 Chapter 6: Santos et al. 2015 (J) 81
- 82. External citations Own citations chapter 228 7 chapter 3 14 5 chapter 4 9 4 chapter 5 10 0 chapter 6 0 0 other co-authored publications 199 48 82
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89. 8 May 2013| @svencharleer | svencharleer.com http://ceur-ws.org/Vol-1103/paper5.pdf89
- 90.
- 91. “What is researchbut a blind date with knowledge?” Will Harvey 91
- 92. Thank you foryour attention! Looking forward to answer your questions. 92