Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Collecting, measuring, analysing and using student data in open distance/distributed learning

302 views

Published on

Unisa UNESCO Chair on Open Distance Learning Series,
15 June 2019, University of South Africa

Published in: Education
  • Be the first to comment

Collecting, measuring, analysing and using student data in open distance/distributed learning

  1. 1. Collecting, measuring, analysing and using student data in open distance/distributed learning Image credit: https://www.flickr.com/photos/themonk/5948439988 Unisa UNESCO Chair on Open Distance Learning Series, 15 June 2019, University of South Africa By Paul Prinsloo 14prinsp
  2. 2. Acknowledgement I do not own the copyright of any of the images in this presentation. I therefore acknowledge the original copyright and licensing regime of every image used. This presentation (excluding the images) is licensed under a Creative Commons Attribution 4.0 International License.
  3. 3. Collecting data in higher education: scope Higher education institution Pre- enrolment/ enquiries/ visits Enrolment Demographic data Pre-entry academic data Engagement with student advising/ counselling/ finance/ faculty/ library Courses Learning management system Engagement with student advising/ counselling/ finance/ faculty/ library Re- enrolment/ enquiries/ visits Post- enrolment From pre-enrolment to re/post-enrolment we collect data/they share data
  4. 4. We are not the only ones collecting, measuring, analysing student data
  5. 5. The world of (student) data Academic Analytics Learning Analytics (Higher) Education • Individuals • Corporates • Governments • Data brokers • Fusion centers • Directed • Automated • Gifted
  6. 6. How/why is the collection, measurement, analysis and use of student data in distance/online/distributed learning different/more important(?) than in residential/traditional higher education?
  7. 7. Distance education institution The impact of distance: • Place • Space • Time • Emotional • Flexibility/ • choice Fully offline/ digitally supported/ internet supported/ internet dependent/ fully online* * Department of Higher Education and Training. (2014). Policy for the provision of distance education in South African universities in the context of an integrated post-school system. Retrieved from https://www.gov.za/ss/documents/higher-education-act-policy- provision-distance-education-south-african-universities Pedagogy • Teaching period: 10-14 weeks • Formative assessment? • Summative assessment • Guided pedagogic conversation What data do we have/need to inform our teaching and support/evaluate students’ learning?
  8. 8. Image credit: https://www.flickr.com/photos/codnewsroom/25137910512 What do you/we know about (y)our students?
  9. 9. Image credit: https://pixabay.com/photos/bboy-breakdance-dancing-hip-hop-1450053/ Who at the institution, knows what about (y)our students? How is this shared? For what purposes? And if not, why not?
  10. 10. What don’t you/we know about (y)our students? Image credit: https://pixabay.com/photos/street-africa-ghana-city-streets-3644374/
  11. 11. What do you/we know about • How they learn • What times of the day they engage with their studies • How often and for what purpose they engage with peers/faculty, etc. Image credit: https://pixabay.com/photos/train-wagon-people-the-crowd-feet-2373323/
  12. 12. Why do you/we want/need to know about them, their lives, their learning journeys? Image credit: https://pixabay.com/photos/red-nose-color-splatter-joy-women-1675188/
  13. 13. Do they know you/we are collecting, measuring, analysing their data and using that data to evaluate and support their learning? Imagecredit:https://pixabay.com/photos/sculpture-bronze-the-listening-2275202/
  14. 14. And under what conditions are the collection, measurement, analysis and use of student data …
  15. 15. And if you know, what can/will you do about the challenges they face? Image credit: https://pixabay.com/photos/train-station-transportation-people-691176/
  16. 16. Image credit: https://www.deviantart.com/kovangfx/art/Being-alone-526166616 What information do they have, that if they had a way or a reason to share it with you/us, this will help you/us to understand their learning journeys better, support them (more) appropriately, and evaluate more fairly?
  17. 17. #Recap 1. What student (learning) data do we/you currently have? 2. Where/how are these data stored, in which formats, for what purpose were/are they collected, by whom and who has access to these data for what purposes? 3. What data don’t we/you have, that if we/you have access to these data, if will help us/you to teach better? 4. What data do we/you currently use to teach better and make more informed decisions, and if not, why not? 5. What data do students need to learn better, more make more appropriate decisions? 6. What data do they have, that, if they would share it with us, can help us to teach better and make more appropriate support decisions?
  18. 18. Overview of the (rest of the)presentation 1. Mapping the evolution in the use of student data 2. How we use student data 3. Who/what is doing the collection, measurement, analysis of and use student data? 4. How does this scale? The potential of algorithmic-decision making systems 5. Making sense of my students’ data: two case studies 6. Using student data as surveillance: concerns and ethical issues 7. Some principles to consider 8. (In)conclusions
  19. 19. (Higher) education has always collected, analysed and used student data – so what has changed? Image credit: https://upload.wikimedia.org/wikipedia/commons/7/79/A_Medieval_Classroom.jpg
  20. 20. Source credit: https://tekri.athabascau.ca/analytics/ “Learning institutions and corporations make little use of the data learners "throw off" (sic) in the process of accessing learning materials, interacting with educators and peers, and creating new content” (emphasis added)
  21. 21. Source credit: https://tekri.athabascau.ca/analytics/ “Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs”.
  22. 22. Source credit: https://er.educause.edu/articles/2011/9/penetrating-the-fog-analytics-in-learning-and-education
  23. 23. IN THE PAST AT PRESENT Data sources Demographic and learning data at specific points in the learning journey: data application, registration, class registers, assignments, summative assessment, personal communication Continuous, directed, gifted and automated collection of data from a range of data sources – student administration, learning management system (LMS), sources outside of the LMS Data use Reporting purposes, operational planning on cohort, group level by management, institutional researchers Descriptive, diagnostic, predictive and prescriptive on group/cohort level Plus individualised, often real- time use of data to inform pedagogy, curriculum, assessment, student support by faculty, students and support staff Who used the data (officially)? Management, institutional researchers, planners, quality assurance and HR departments Plus researchers, faculty, students and support staff
  24. 24. IN THE PAST AT PRESENT Who did the collection, analysis and who used the data Humans Increasingly humans in combination with algorithmic decision-making processes Temporal aim Retrospective/historical data to make predictions with regard to budget, future enrollments & resource allocation on institutional level Plus real-time data for real-time interventions Default Forgetting Remembering Personal identifiers Anonymised, aggregated data Plus re-identifiable data Personal/ised data Oversight/ data governance Broad institutional oversight. Ethical Review Board (ERB) approval for research purposes Approval, oversight and governance highly complex and contested
  25. 25. We know, take into account and we measure: age, gender, race, street address and zip code, occupation, pre-enrolment educational data, registration data, engagement data, academic data, library data, financial aid data, behavioural data, location data, who-are-in-their-networks-data, their chances of failing, dropping out, stopping out… Image credit: https://pixabay.com/en/side-profile-black-male-student-1440176/ And we use this data to… Image credit: https://pixabay.com/en/girl-library-education-student-1721436/
  26. 26. Source credit: http://timoelliott.com/blog/2013/02/gartnerbi-emea-2013-part-1-analytics-moves-to-the-core.html
  27. 27. Students (with a particular habitus and demographics) apply to register, choose a qualification and modules, and start… Image credit: https://pixabay.com/en/side-profile-black-male-student-1440176/ Image credit: https://pixabay.com/en/girl-library-education-student-1721436/ Macro-societal factors, e.g. economic, political, social, technological, environmental and legal factors. Institutional/lecturer inactions, or inefficiencies impacting and shaping their behavioral data, their chances of failing, dropping out, stopping out…
  28. 28. Image credit: https://pixabay.com/en/side-profile-black-male-student-1440176/ Image credit: https://pixabay.com/en/girl-library-education-student-1721436/ Macro-societal factors, e.g. economic, political, social, technological, environmental and legal factors. Institutional/lecturer inactions, or inefficiencies impacting and shaping their behavioral data, their chances of failing, dropping out, stopping out… Data points
  29. 29. Processes Inter & intra- personal domains Modalities: • Attribution • Locus of control • Self-efficacy Processes Modalities: • Attribution • Locus of control • Self-efficacy Domains Academic Operational Social TRANSFORMED INSTITUTIONAL IDENTITY & ATTRIBUTES THE STUDENT AS AGENT IDENTITY, ATTRIBUTES, HABITUS Success THE INSTITUTION AS AGENT IDENTITY, ATTRIBUTES, HABITUS SHAPING CONDITIONS: (predictable as well as uncertain) SHAPING CONDITIONS: (predictable as well as uncertain) Choice, Admission Learning activities Course success Gradua- tion THE STUDENT WALK Multiple, mutually constitutive interactions between student, institution & networks F I T FIT F I T FIT Employ- ment/ citizenship TRANSFORMED STUDENT IDENTITY & ATTRIBUTES F I T F I T F I T F I T F I T F I T F I T F I T Retention/Progression/Positive experience
  30. 30. Source credit: https://www.tandfonline.com/doi/abs/10.1080/01587919.2011.584846
  31. 31. Image credit: https://pixabay.com/en/side-profile-black-male-student-1440176/ Image credit: https://pixabay.com/en/girl-library-education-student-1721436/ Macro-societal factors, e.g. economic, political, social, technological, environmental and legal factors. Institutional/lecturer inactions, or inefficiencies impacting and shaping their behavioral data, their chances of failing, dropping out, stopping out… Data points
  32. 32. Image credit: https://pixabay.com/illustrations/technology-programming-binary-robot-2062712/ Who/what is doing the collection, measurement, analysis and how are these analyses implemented and used? Collect/measure/analyse • Institutional analysts • Researchers • Faculty • Support staff • Administrators • Management • Algorithms Use • Institutional analysts • Researchers • Faculty • Support staff • Administrators • Management • Students • Algorithms
  33. 33. Image credit: https://pixabay.com/illustrations/technology-programming-binary-robot-2062712/ In an open distance and distributed learning environment, with thousands of students registered for any particular module, how does this scale? Collect/measure/analyse • Institutional analysts • Researchers • Faculty • Support staff • Administrators • Management • Algorithms Use • Institutional analysts • Researchers • Faculty • Support staff • Administrators • Management • Students • Algorithms
  34. 34. (1) Humans perform the task (2) Task is shared with algorithms (3) Algorithms perform task: human supervision (4) Algorithms perform task: no human input Seeing Yes or No? Yes or No? Yes or No? Yes or No? Processing Yes or No? Yes or No? Yes or No? Yes or No? Acting Yes or No? Yes or No? Yes or No? Yes or No? Learning Yes or No? Yes or No? Yes or No? Yes or No? Danaher, J. (2015). How might algorithms rule our lives? Mapping the logical space of algocracy. [Web log post]. Retrieved from http://philosophicaldisquisitions.blogspot.com/2015/06/how-might-algorithms-rule-our-lives.html Can Artificial Intelligence (AI) and algorithmic decision- making systems help?
  35. 35. Image credit: https://pixabay.com/photos/ceramic-clay-pottery-art-old-3050615/ Working with data: a personal journey
  36. 36. CASE STUDY 1: Fully Online (structured) German Masters degree Pedagogy • 15 weeks • 2 instructors, one support staff, 20 students • Weekly structure with assigned readings, activities, discussion forum requirement • In total: 3 tasks, 2 essays, 6 skill builder exercises, group work, structured learning journal • Students expected to actively participate in the class – at least one substantive posting per week, plus respond in a substantive way to other students’ posts Student data • Gender • Employment • Educational background (to some extent) • Last 20 logins • Gifted information (“I hurt my back last week”)
  37. 37. Male Mid-thirties Little to no engagement Average assignment Female, mid-thirties Contributes frequently Excellent assignments Female Late forties Little to no engagement Failed first assignment
  38. 38. Male Mid-thirties Little to no engagement Average assignment Female, mid-thirties Contributes frequently Excellent assignments Female Late forties Little to no engagement Failed first assignment
  39. 39. While I did not know what it meant, it did not prevent me from responding… Email sent on 23 March in response to no activity for 5 days
  40. 40. Email sent on 31 March in response to no activity for 5 days
  41. 41. Reflection: Case Study 1 • The learning/pedagogy design/engagement is immensely intensive for both instructors and students • Students could not afford to fall behind • Students found the high level of engagement and amount of readings overwhelming • The only data I had access to was their last 20 logins, and information they ‘gifted’ • I wish some form of algorithmic automated system could have alerted me when students did not login for three days • While having access to login data proved immensely helpful in the context of this course, looking for the data and responding to the data were also very taxing and time-consuming • Yes, the instructor: student ratio of 1:20 made a huge difference
  42. 42. CASE STUDY 2: Fully Online (structured) US Masters degree Pedagogy • 15 weeks • 2 instructors, one support staff, 20-25 students • Weekly structure with assigned readings, activities, discussion forum requirement • In total: 3 tasks, 2 essays, 6 skill builder exercises, group work, structured learning journal • Students expected to actively participate in the class – at least one substantive posting per week, plus respond in a substantive way to other students’ posts Student data • Gender • Employment • Educational background (to some extent) • A range of login data – number, time-on-task, responses, scores, progress, etc • Gifted information (“My partner lost his job”)
  43. 43. Exhibit A
  44. 44. Exhibit B
  45. 45. Image credit: https://pixabay.com/en/person-listening-headphones-eyes-2157916/ What does this mean?
  46. 46. Exhibit C
  47. 47. Exhibit D
  48. 48. Prinsloo, P. (2016, November 7 ). Failing our students: not noticing the traces they leave behind. [Web log post]. Retrieved from https://opendistanceteachingandlearning.wordpress.com/2016/11/07/failing-our-students-not-noticing-the-traces-they-leave- behind/ Reflection: Case Study 2
  49. 49. Image credit: https://pixabay.com/en/buddha-statue-stone-statues-speak-546458/ If these data tell a story, whose story is it? Is this the whole story? Who is listening? What do we miss?
  50. 50. Collecting student data: some concerns Image credit: https://pixabay.com/photos/monitoring-safety-surveillance-1305045/
  51. 51. Source credit: https://www.washingtonpost.com/news/the-switch/wp/2015/12/28/google-is-tracking-students- as-it-sells-more-products-to-schools-privacy-advocates-warn/?noredirect=on&utm_term=.6c0bfa947fc2
  52. 52. Source credit:https://www.insidehighered.com/news/2018/10/09/coming-soon-ivy-league-campuses-free-coffee-privacy-not-included
  53. 53. Source credit: https://www.theguardian.com/higher-education-network/2016/aug/03/learning-analytics-universities-data-track-students 2016
  54. 54. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/ Page credit: https://www.edsurge.com/news/2017-06-13-from-high-school-to-harvard-students-urge-for-clarity-on-privacy- rights?utm_content=buffer8dd71&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer 3 June 2017
  55. 55. Source credit: https://campustechnology.com/articles/2018/05/02/when-learning-analytics-violate-student-privacy.aspx 2018
  56. 56. Image credit: https://pixabay.com/en/travel-sculpture-man-stone-move-3034459/ Knowing more also means having greater responsibility to make informed, tentative, appropriate and ethical decisions
  57. 57. Ethics in learning analytics: Selected examples 2013-2017
  58. 58. Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American Behavioral Scientist 57(1), 1509–1528.
  59. 59. Source credit: https://www.open.ac.uk/students/charter/sites/www.open.ac.uk.students.charter/files/files/ecms/web-content/ethical- use-of-student-data-policy.pdf 2014
  60. 60. Source credit: https://www.open.ac.uk/students/charter/sites/www.open.ac.uk.students.charter/files/files/ecms/web-content/ethical- use-of-student-data-policy.pdf Principle 1: Learning analytics is an ethical practice that should align with core organisational principles, such as open entry to undergraduate level study. Principle 2: The OU has a responsibility to all stakeholders to use and extract meaning from student data for the benefit of students where feasible. Principle 3: Students should not be wholly defined by their visible data or our interpretation of that data. Principle 4: The purpose and the boundaries regarding the use of learning analytics should be well defined and visible.
  61. 61. Source credit: https://www.open.ac.uk/students/charter/sites/www.open.ac.uk.students.charter/files/files/ecms/web-content/ethical- use-of-student-data-policy.pdf Principle 5: The University is transparent regarding data collection, and will provide students with the opportunity to update their own data and consent agreements at regular intervals. Principle 6: Students should be engaged as active agents in the implementation of learning analytics (e.g. informed consent, personalised learning paths, interventions). Principle 7: Modelling and interventions based on analysis of data should be sound and free from bias. Principle 8: Adoption of learning analytics within the OU requires broad acceptance of the values and benefits (organisational culture) and the development of appropriate skills across the organisation.
  62. 62. Willis, J. E., Slade, S., & Prinsloo, P. (2016). Ethical oversight of student data in learning analytics: A typology derived from a cross-continental, cross-institutional perspective. Educational Technology Research and Development, 64, 881-901. DOI: 10.1007/s11423-016-9463-4 http://link.springer.com/article/10.1007/s11423- 016-9463-4 Who will provide oversight over the ethical issues in learning analytics? An interpretative multiple-case study: Indiana University, Open University (UK) and the University of South Africa (Unisa) 2016
  63. 63. Source credit: https://www.siyaphumelela.org.za/documents/5a61c7b737ff5.pdf 2017
  64. 64. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/ Guiding principles for an ethics of care: Principle 1: The moral, relational duty of learning analytics Principle 2: Defining student success in the nexus of student, institution and macro-societal agencies and context Principle 3: Understanding data as framed and framing Principle 4: Student data sovereignty Principle 5: Accountability Principle 6: Transparency Principle 7: Co-responsibility
  65. 65. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/ Prinsloo, P. (2018). Include us all! Directions for the adoption of learning analytics in the Global South: An African perspective. Digital Learning for Development (DL4D). Available at: http://dl4d.org/portfolio- items/learning-analytics-for-the-global-south/
  66. 66. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/ Image credit: https://pixabay.com/photos/compass-hand-travel-direction-1753659/ Some guiding principles
  67. 67. “If you have come to help us, you can go home. If you have come to accompany us, please come. We can talk” Glesne, C. (2016). Research as solidarity. In T. Kukutai and J. Taylor. (Eds), Indigenous data sovereignty. Toward an agenda (pp. 169-178). Canberra, Australia: Australian National University Press. Retrieved from https://press.anu.edu.au/publications/series/centre-aboriginal-economic-policy- research-caepr/indigenous-data-sovereignty Principle 1: The moral, relational duty of learning analytics Image credit: https://pixabay.com/en/sculpture-bronze-2196139/
  68. 68. Principle 2: Student success as entangled Image credit: https://pixabay.com/en/rope-knot-string-strength-cordage-3052477/
  69. 69. THE STUDENT AS AGENT IDENTITY, ATTRIBUTES, HABITUS Success THE INSTITUTION AS AGENT IDENTITY, ATTRIBUTES, HABITUS SHAPING CONDITIONS: (predictable as well as uncertain) SHAPING CONDITIONS: (predictable as well as uncertain) THE STUDENT WALK Multiple, mutually constitutive interactions between student, institution & networks FIT FIT F I T F I T F I T F I T F I T F I T F I T F I T Subotzky, G., & Prinsloo, P. (2011). Turning the tide: a socio-critical model and framework for improving student success in open distance learning at the University of South Africa. Distance Education, 32(2): 177-19.
  70. 70. Principle 3: Understanding data as broken and framed but also as acting and framing Image credit: https://pixabay.com/en/eyeglasses-broken-glasses-sight-366446/
  71. 71. Pink, S., Ruckenstein, M., Willim, R., & Duque, M. (2018). Broken data: Conceptualising data in an emerging world. Big Data & Society, 5(1), 2053951717753228. We have to recognize that our data are broken
  72. 72. “Data is not necessarily accurate, complete or full aggregated representations of what individuals or societal groups have done, or able to predict what they will do” (Pink et al., 2018, p. 10) Pink, S., Ruckenstein, M., Willim, R., & Duque, M. (2018). Broken data: Conceptualising data in an emerging world. Big Data & Society, 5(1), 2053951717753228.
  73. 73. Student data are not something separate from students’ identities, their histories, their beings. Data are an integral, albeit informational part of students being. In the light of the view that data are not something students own but rather who they are; what are we assuming when we say we ‘collect’ their data? E.g. Floridi, L. (2005). The ontological interpretation of informational privacy. Ethics and Information Technology, 7(4), 185-200. Principle 4: Student data sovereignty
  74. 74. Ctrl Alt Del • How much control do students have to determine what data institutions harvest; to challenge the meaning of their data and our categories of analysis? • Can students (re)define/alter interpretations of data and definitions/categories? Can they offer us counter- narratives to our understanding of their learning and their life-worlds? • Can students opt out of personalised data collection, analysis and use and have their data deleted?
  75. 75. Willis, J. E., Slade, S., & Prinsloo, P. (2016). Ethical oversight of student data in learning analytics: A typology derived from a cross- continental, cross-institutional perspective. Educational Technology Research and Development, 64, 881-901. DOI: 10.1007/s11423- 016-9463-4 http://link.springer.com/article/10.1007/s11423-016-9463-4 Principle 5: Accountability An interpretative multiple-case study: Indiana University, Open University (UK) and the University of South Africa (Unisa)
  76. 76. Principle 6: Transparency If they don’t know that we collect their data, the scope and purpose of the collection, how we will use their data and how it will impact on their learning journeys, how is this ethical? Image credit: https://pixabay.com/en/sculpture-bronze-the-listening-2209152/
  77. 77. Principle 7: Co-responsibility Our students’ journeys are intimately weaved into our (institutional) stories. What happens in their lives, impact ours. And vice versa. Image credit: https://pixabay.com/en/weave-hand-labor-samoa-exotic-55/
  78. 78. Student data are an invitation to start a conversation Image credit: https://pixabay.com/photos/man-sculpture-art-wonders-talk-1483479/ Image credit: https://pixabay.com/photos/man-sculpture-art-wonders-talk-1483479/ Principle 8: Student data as conversation
  79. 79. Principle 9: You cannot unknow
  80. 80. Prinsloo, P., & Slade, S. (2017, March). An elephant in the learning analytics room: the obligation to act. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 46-55). ACM.
  81. 81. Prinsloo, P., & Slade, S. (2014). Educational triage in higher online education: walking a moral tightrope. International Review of Research in Open Distributed Learning (IRRODL), 14(4), 306-331. http://www.irrodl.org/index.php/irrodl/article/view/1881
  82. 82. Image credit: https://pixabay.com/en/stairs-boots-old-shoes-1683118/ Collecting, analysing and using student data may assist them and us, to make more appropriate, and responsible choices
  83. 83. Imagecredit:https://pixabay.com/en/sculpture-bronze-child-boy-1392529/ Thank you Paul Prinsloo (Prof) Research Professor in Open Distance Learning (ODL), University of South Africa, Unisa, T: +27 (0) 12 433 4719 (office) prinsp@unisa.ac.za Skype: paul.prinsloo59 Personal blog: http://opendistanceteachingandlearning.wordpress.com Twitter profile: @14prinsp

×