Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to What can learning analytics do for us
Similar to What can learning analytics do for us (20)
More from Yi-Shan Tsai
Recently uploaded
Recently uploaded (20)
What can learning analytics do for us
- 1. What can learning analytics do for us? Yi-Shan Tsai yi-shan.tsai@ed.ac.uk @yi_shan_tsai http://sheilaproject.eu/ RTEN seminar series 30 January 2019
- 2. What to expect… • What is learning analytics (LA) • Project overview • Multi-stakeholder viewpoints on the pros and cons • Implications for a learning analytics (LA) strategy • SHEILA framework Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://sheilaproject.eu/
- 3. What is learning analytics?
- 5. Learning analytics is… “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” (Long et al., 2011). Long, P. D., Siemens, G., Conole, G., & Gašević, D. (Eds.). (2011). In Proceedings of the 1st International Conference on Learning Analytics and Knowledge (LAK’11). Banff, AB, Canada: ACM. http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
- 6. Learning analytics is… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Image attribution: http://sheilaproject.eu/
- 7. Learning analytics examples http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
- 8. Learning analytics examples http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Arnold, K. E., & Pistilli, M. D. (2012, April). Course Signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 267-270). Goal: address retention Predictive algorithm: -Performance -Effort -Prior academic history -Student characteristics
- 9. Learning analytics examples http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://edin.ac/2m3vklH
- 10. Learning analytics examples http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://edin.ac/2m3vklH
- 11. Learning analytics examples http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Learning Analytics Report Card (LARC) http://larc-project.com Involve students in critical conversations around the use of their data for computational analysis in education. Knox, J. (2017). Data Power in Education: exploring critical awareness with the ‘Learning Analytics Report Card’ (LARC). Special Issue: Data Power in Material Contexts, Journal of Television and Media. http://journals.sagepub.com/doi/full/10.1177/1527476417690029 Isard, A. and Knox, J. 2016. Automatic Generation of Student Report Cards. 9th International Natural Language Generation conference. Edinburgh, Sept 5-8 http://www.macs.hw.ac.uk/InteractionLab/INLG2016/proceedings/pdf/INLG33.pdf
- 12. Learning analytics examples http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
- 13. Learning analytics is… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Clow, D. (2012, April). The learning analytics cycle: closing the loop effectively. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 134-138). ACM.
- 14. SHEILA project Supporting Higher Education to Integrate Learning Analytics
- 15. Objectives • Understand the state of the art • Engage key stakeholders directly • Develop a policy framework Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://sheilaproject.eu/
- 16. Methodology Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://sheilaproject.eu/
- 17. Challenges in institutional adoption of LA Literature review Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://sheilaproject.eu/ Tsai, Y. S., & Gasevic, D. (2017). Learning analytics in higher education – challenges and policies: a review of eight learning analytics policies. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 233-242).
- 18. 1. Insufficient leadership driving strategic implementation & monitoring Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://sheilaproject.eu/ https://pixabay.com/
- 19. 2. Unequal engagement with primary stakeholders Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://sheilaproject.eu/ https://pixabay.com/
- 20. 3. Gaps in data literacy across stakeholders Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://sheilaproject.eu/ https://pixabay.com/
- 21. 4. Weak grounding of learning theories Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://sheilaproject.eu/ https://pixabay.com/
- 22. 5. Little evidence demonstrating the impacts of LA-based interventions Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://sheilaproject.eu/ https://pixabay.com/
- 23. 6. Lack of institution-based polices for learning analytics practices Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework http://sheilaproject.eu/ https://pixabay.com/
- 26. Managers would like learning analytics to… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework • To improve student learning performance – 40 (87%) • To improve student satisfaction – 33 (72%) • To improve teaching excellence – 33 (72 %) • To improve student retention– 26 (57 %) • To explore what learning analytics can do for our institution/ staff/ students – 25 (54 %) Institutional survey (n=46) Tsai, Y.-S., & Gašević, D. (2017). The State of Learning Analytics in Europe – Executive Summary – SHEILA (Executive summary). Retrieved from http://sheilaproject.eu/2017/04/18/the-state- of-learning-analytics-in-europe-executive-summary/
- 27. Managers would like learning analytics to… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework • To provide personalised learning support (39 %) • To increase learning motivations (37 %) • To inform curriculum (35 %) • To encourage self-regulated learning (30 %) • To improve student-teacher communication (26 %) • To improve student recruitment (24 %) • Other (2 %) Institutional survey (n=46) Tsai, Y.-S., & Gašević, D. (2017). The State of Learning Analytics in Europe – Executive Summary – SHEILA (Executive summary). Retrieved from http://sheilaproject.eu/2017/04/18/the-state- of-learning-analytics-in-europe-executive-summary/
- 28. Institutional goals & approaches • Interview data (27 institutions) • Epistemic Network Analysis (ENA) • The concurrence of codes implies the strength of connection • The frequency of code concurrence is expressed by the weight (thickness) of the lines http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Shaffer, D. W., Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics, 3(3), 9-45.
- 29. Comparison by adoption experience Less than one year One year or more http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
- 30. Teachers would like learning analytics to… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework STUDENT LEVEL TEACHER LEVEL PROGRAM LEVEL Take responsibility for their learning and enhancing their SRL- skills Assess the degree of success to prevent students from begin worried or optimistic about their performance Method to identify student’s weaknesses and know where students are with their progress Understand how students engage with learning content Improve of the design and provision of learning materials, courses, curriculum and support to students Understand how program is working (strengths and bottlenecks) Improve educational quality (e.g. content level) Focus groups: • 16 groups • 4 universities • 59 participants
- 31. Teachers would like learning analytics to… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Focus groups: • 16 groups • 4 universities • 59 participants Know the ‘usefulness’ of resources and the preferences of students towards learning materials Enable personalised support to second language speakers Evaluate the workload of students who were mostly part-time learners UK & Spanish FGs Estonian FGs Dutch FGs
- 32. Students would like learning analytics to… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Personalised support Feedback Academic resources Teaching quality Focus groups: • 18 groups • 4 universities • 74 participants
- 33. Desires for self-regulated learning http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Survey: • 6 institutions • 3053 returns SRL: • Receiving a complete profile of their learning • Making their own decisions based on the analytics results • Knowing how their progress compares to a set learning goal
- 34. High expectations of self-regulated learning http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Survey: • 6 institutions • 3053 returns SRL: • Receiving a complete profile of their learning • Making their own decisions based on the analytics results • Knowing how their progress compares to a set learning goal
- 36. Managers are concerned about… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Returns on investment Resources Culture Skills
- 37. Returns on investment… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework “We could have had a little black box instead of an expensive piece of software. The wrapping around the project, the energy, the commitment, the targeted actions, how much of that would have just delivered some change anyway?” – Manager
- 38. Returns on investment… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework “Learning analytics is at an early stage and so over time it may be able to do a lot more things than we can do now. In fact, it will but there is this problem that if things don’t deliver what people expect quickly, they then say, ‘oh there’s no point in doing that’, and it gets a bad name.” – Manager
- 39. Teachers are concerned about… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Students Teachers Learning analytics
- 40. Teachers are concerned about… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework • Student-level https://www.pinterest.com/pin/432486370448743887/
- 41. Teachers are concerned about… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework • Student-level https://www.pinterest.com/pin/432486370448743887/
- 42. Teachers are concerned about… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework • Student-level https://www.pinterest.com/pin/432486370448743887/
- 43. Teachers are concerned about… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework • Teacher-level https://www.pinterest.com/pin/432486370448743887/
- 44. Teachers are concerned about… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework • Teacher-level https://goo.gl/images/cA7J3h
- 45. Teachers are concerned about… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework • Learning analytics-level https://goo.gl/images/TK6J8p Can individual differences be captured?
- 46. Teachers are concerned about… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework • Learning analytics-level https://goo.gl/images/TVLukx Interpretations of learning vary
- 47. Teachers are concerned about… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework “I don’t want it [LA] to make all of the students behave in the exact same way to satisfy an algorithm. I want it to enable students to have the best experience in whatever that experience is. You know, you can be totally different from everyone else and still do perfectly fine. I want it [LA] to…enable students to do better and not make them all mini ‘me’s.” - Teacher
- 48. Students are concerned about… http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework Privacy Stereotypes Learning being quantified Losing human contacts
- 49. http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework “Although consumers seem to be concerned about their privacy as reflected in their intentions to disclose (e.g., measured via ‘‘willingness to provide information’’), anecdotal evidence suggests their behaviors diverge from their intentions to disclose personal details.” (Norberg, Horne, & Horne, 2007, p. 107) Privacy paradox
- 50. 1. Perceived benefits outweigh perceived risks http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework “I haven’t been in University for so long, so for me to get back to the school was challenging, and so for me with the personal tutor I don’t mind sharing my data ‘cause she will help me to develop myself further.” - Student
- 51. 2. Power imbalance http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework “You have to agree to share this data otherwise you wouldn’t enrol, so you are not probably thinking that much about consequences of every single piece of data that you provide to the university. It’s just because it’s a part of the process of application.” - Student
- 52. 3. Trust exacerbates information asymmetry http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework “That [Data policy] is something I don’t think I would ever focus on or look for, so I honestly don’t know. It could be out there and I could maybe Google it, and it would be on a page somewhere if I wanted to find it.” - Student
- 53. Implications for LA strategy http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
- 54. 1. A sound policy and effective communication Purpose, access, anonymity, and security http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
- 55. 2. Increase the observability and trialability of LA to attract buy-in An incremental approach http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
- 56. 3. Install data literacy and reflective skills among key stakeholders Moving from data to action http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
- 57. 4. Incorporating the views of different stakeholders to develop a common vision and a sense of ownership A dialogic approach http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
- 58. 5. Clarify the value of LA within its limitations Expectation management http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
- 59. SHEILA framework Action, challenges, and policy http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
- 60. http://sheilaproject.eu/sheila-framework/ Tsai, Y. S., Moreno-Marcos, P. M., Jivet, I., Scheffel, M., Tammets, K., Kollom, K., & Gašević, D. (2018). The SHEILA Framework: Informing Institutional Strategies and Policy Processes of Learning Analytics. Journal of Learning Analytics, 5(3), 5-20. http://sheilaproject.eu/ Learning analytics (LA) SHEILA project Stakeholder views Implications for LA strategy SHEILA framework
Editor's Notes
- Learning today often takes place in a hybrid environment – a combination of online and offline settings. In a traditional classroom, we observe how students are doing with their studies through direct interactions in the class and evaluations of their assignments or exams. Today, the shift of learning to the online setting means classroom observations are no longer sufficient. Meanwhile, the advancement of technology means it is possible for us to collect a wide range of digital data from students’ interactions with the learning environments, e.g., log-ins, access time, length of time spent. Learning analytics analyses the data by looking for patterns.
- A 3-year pilot project with Civitas Learning started in 2016 to investigate student engagement in the digital learning environments. The project serves a strategic purpose to gain experience of developing learning analytics models and develop a Learning Analytics Policy.
- Signals is based on predictive models. The purpose is to determine in real time which students might be at risk, partially indicated by their effort within a course. Interventions may be: Posting of a traffic signal indicator on a student’s LMS home page; • E-mail messages or reminders; • Text messages; • Referral to academic advisor or academic resource centers; or, • Face to face meetings with the instructor.
- A 3-year pilot project with Civitas Learning started in 2016 to investigate student engagement in the digital learning environments. The project serves a strategic purpose to gain experience of developing learning analytics models and develop a Learning Analytics Policy.
- To close the feedback loop
- The project does not deal with technical issues but socio-cultural issues around LA.
- Following the literature review, we carried out a series of research activities to get a better understanding of the identified challenges and other factors that influence the adoption of LA.
- Following the literature review, we carried out a series of research activities to get a better understanding of the identified challenges and other factors that influence the adoption of LA.
- Following the literature review, we carried out a series of research activities to get a better understanding of the identified challenges and other factors that influence the adoption of LA.
- Following the literature review, we carried out a series of research activities to get a better understanding of the identified challenges and other factors that influence the adoption of LA.
- Following the literature review, we carried out a series of research activities to get a better understanding of the identified challenges and other factors that influence the adoption of LA.
- Following the literature review, we carried out a series of research activities to get a better understanding of the identified challenges and other factors that influence the adoption of LA.
- Following the literature review, we carried out a series of research activities to get a better understanding of the identified challenges and other factors that influence the adoption of LA.
- A survey question (multiple choices) provided 11 options for motivations specific to learning and teaching. 46 from 22 countries responded (response rate: 15%). The top 4 are strongly associated with institutional KPIs
- By contrast, only the ‘improving student recruitment’ option is more directly linked to institution KPIs. Perhaps managers do not see a close link between this and LA.
- Institutional goal: Goals set for LA are to improve institutional performance or management. LA will influence decisions made by senior managers. Teaching goal: Goals set for LA are to inform teaching and support. LA is to will influence decisions made by teachers. Learning level: Goals set for LA are to improve learning and student experience. LA is to will influence decisions made by students. These connections suggest that institutions or projects that set out to improve institutional performance through LA tended to have identified clear problems to solve, e.g., student retention issues. By contrast, institutions or projects that aimed to use LA to inform teaching and student support had a stronger focus on exploring a phenomenon, e.g., how students engage with learning resources.
- For novice institutions, LA was often adopted as a measuring tool for institutional performance, e.g., student retention rate. By contrast, more experienced institutions showed strong connections between teaching-level goals and exploratory approaches . Though institutional goal dominates, there is a shift of focus towards teaching and learning.
- Provision of timely feedback, easy access to digital resources, and personalised learning support
- Even though the average responses tended to be similar across locations, the sample of students from the Open University of the Netherlands were found to have lower ideal expectations towards receiving complete profiles across modules based on LA, compared to the other samples. This highlights that there is no one-size-fits-all LA solution, and further investigation into the preference of students towards the access to their learning data at this particular institution is needed.
- U19
- U1
- How data is collected, analysed and interpreted affects our intrepretations of learning
- Students expressed protective attitudes of personal data, but the described actions that they’ve taken to protect their personal data showed the contrary.
- Trust plays a role here.
- Five attributes of innovations: relative advantage, compatibility, complexity, trialability, observability - Rogers, Everett. M. (2010). Diffusion of innovations. Simon and Schuster.