Download to read offline
Uploaded byJisc
Data and disadvantaged students - using learning analytics for inclusion
The document summarizes a presentation on using learning analytics to support disabled students. It discusses ethical issues around collecting student data and potential scenarios for how data could help various stakeholders, such as enabling disability services and exam officers to better support students. It also describes a case study where odds ratios were used to identify modules with accessibility deficits by looking at completion rate disparities between disabled and non-disabled students. The presentation concludes by discussing possibilities for future work, such as using analytics to identify "critical learning paths" and accessibility issues in individual modules.
More Related Content
Similar to Data and disadvantaged students - using learning analytics for inclusion
Data and disadvantaged students - using learning analytics for inclusion
- 1. Niall Sclater, Consultant Dataand disadvantaged students - using learning analytics for inclusion27/02/2017
- 2. While you wait… http://tiny.cc/data-form »Ifyou haven’t had the chance to do so, please take some time to look at the data and disadvantaged students pre- session googleform. 02/03/2017 Data and disabled students 2
- 3. The net ofmeanings - 1 02/03/2017 Data and disabled students 3
- 4. The net ofmeanings - 2 02/03/2017 Data and disabled students 4
- 5. In this sessionwe will… Explore : »Ethical issues »Disability definitions and consequences »Potential scenarios »Your priorities »A real life case study 02/03/2017 Data and disabled students 5
- 6. “learning analytics isthe measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” SoLAR – Society for Learning Analytics Research 02/03/2017 Data and disabled students 6
- 7. LearningAnalytics Service Toolkit Community Jisc LearningAnalyticsProject Jisc Learning Analytics 2017
- 8. 86 issues in9 groups Group Name Question Main type Importance Responsibility 2 Consent Adverse impact of opting out on individual If a student is allowed to opt out of data collection and analysis could this have a negative impact on their academic progress? Ethical 1 Analytics Committee 7 Action Conflict with study goals What should a student do if the suggestions are in conflict with their study goals? Ethical 3 Student 8 Adverse impact Oversimplification How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances? Ethical 1 Educational researcher
- 9. Group Name QuestionMain type Importance Responsibility 2 Consent Adverse impact of opting out on individual If a student is allowed to opt out of data collection and analysis could this have a negative impact on their academic progress? Ethical 1 Analytics Committee 7 Action Conflict with study goals What should a student do if the suggestions are in conflict with their study goals? Ethical 3 Student 8 Adverse impact Oversimplification How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances? Ethical 1 Educational researcher jisc.ac.uk/guides/code-of-practice-for-learning- analytics
- 10.
- 11.
- 12. Accessibility Considerations forLearning Analytics 1. Remember that learning analytics is not assessment 2. Avoid the labelling of individuals and reinforcing of prejudice and stereotypes 3. Maintain disabled students’ confidentiality 4. Handle the inference of disabilities from the analytics appropriately 5. Ensure that the analytics do not unfairly single out disabled students 6. Use analytics to identify modules where there appear to be accessibility issues 7. Ensure that student-facing analytics are accessible 8. Ensure that interventions are worded appropriately » https://analytics.jiscinvolve.org/wp/2016/12/14/accessibility-considerations-for-learning-analytics/ Jisc Learning Analytics Accessibility Webinar
- 13. Contacts Paul Bailey paul.bailey@jisc.ac.uk NiallSclater niall.sclater@jisc.ac.uk Further Information: http://www.analytics.jiscinvolve.org Join: analytics@jiscmail.ac.uk Jisc Learning Analytics 2017
- 14. Straw poll 02/03/2017 Dataand disabled students 14
- 15. The data discussion 1.How do you encourage disabled students to disclose? 2. Which of your institutional data sources might be relevant to supporting disabled students? 3. Which disabled students are visible in your data? 4. How mature is the technology? Discussion: 02/03/2017 Data and disabled students 15
- 16. Brainstorm Scenarios » Designof learner data › What is collected (ethics, level of detail e.g. disability and other co-morbid factors, process and outcome), › Design of interface (usability and accessibility), » Support learner progress › Documenting/disclosing barriers provides an additional method of early identification for support by tracking progress, › Informing course design and learner attainment › Improving learning and teaching practice › Comparing progress of disabled versus non disabled learners » Evaluate institutional/support services › Usage of institution wide assistive technology (e.g. text to speech) › Library uptake of productivity software, ebook usage, 02/03/2017 Data and disabled students 16
- 17. Design: Stakeholder engagement: Seniormanager »“We've invited a range of stakeholders to be involved in our learning analytics steering group – including support staff and people with accessibility needs” 02/03/2017 Data and disabled students 17 Roll of roles… Senior manager
- 18. Design: Learner engagement StudentUnion President »“Students are actively involved in deciding what information they see on their own personal dashboard.” 02/03/2017 Data and disabled students 18 Roll of roles… Senior manager Student union president
- 19. Design: Exam arrangements ExaminationOfficer »“Some people need access arrangements for exams. It’s always involved lots of consultation and meetings. Now its just a touch of a button” 02/03/2017 Data and disabled students 19 Roll of roles… Senior manager Student union president Examination officer
- 20. Support: Consistency Study skillstutor »Now there’s less room for students to slip through the net because all the support services have the same information at the same time so can work together. 02/03/2017 Data and disabled students 20 Roll of roles… Senior manager Student union president Examination officer Study skills tutor
- 21. Support: Responsiveness Head ofdisability service » Student data helps me track the progress of students who have disclosed a barrier to learning so we can respond more swiftly. 02/03/2017 Data and disabled students 21 Roll of roles… Senior manager Student union president Examination officer Study skills tutor Head of disability service
- 22. Support: Prioritise Dyslexia specialist »I used to spend ages chasing students who missed their appointments. Now I can instantly check their other progress and leave them alone if they’re succeeding. 02/03/2017 Data and disabled students 22 Roll of roles… Senior manager Student union president Examination officer Study skills tutor Head of disability service Dyslexia specialist
- 23. Evaluate: Support strategies Assistivetechnologist »As well as asking about a learners disability we've tried to capture more specific detail about the technology strategies recommended through the assessment. 02/03/2017 Data and disabled students 23 Roll of roles… Senior manager Student union president Examination officer Study skills tutor Head of disability service Dyslexia specialist Assistive technologist
- 24. Evaluate:Teaching approaches Lecturer »I cansee how changes to my resources and activities have impacted on everyone's engagement, and particularly benefited my disabled students. 02/03/2017 Data and disabled students 24 Roll of roles… Senior manager Student union president Examination officer Study skills tutor Head of disability service Dyslexia specialist Assistive technologist Lecturer
- 25. Evaluate:Teaching approaches Learning technologist »“Ican see who is using the learning platform and how often .This makes it easy to see where content might be difficult to access.” 02/03/2017 Data and disabled students 25 Roll of roles… Senior manager Student union president Examination officer Study skills tutor Head of disability service Dyslexia specialist Assistive technologist Lecturer Learning technologist
- 26. Evaluate: Library technologysupport Library manager » We can now monitor the usage/uptake of enabling technology software in our library. This helps us to adopt a more targeted strategy for promotion of productivity tools to enhance the support we offer. 02/03/2017 Data and disabled students 26 Roll of roles… Senior manager Student union president Examination officer Study skills tutor Head of disability service Dyslexia specialist Assistive technologist Lecturer Learning technologist Library manager
- 27. Evaluate: E-books andjournals: Collections manager »“I can see who is using the resources such as e-books and e-journals. If there are any anomalies I can ask why. By exploring use by different categories of student we can plan more effective intervention and support” 02/03/2017 Data and disabled students 27 Roll of roles… Senior manager Student union president Examination officer Study skills tutor Head of disability service Dyslexia specialist Assistive technologist Lecturer Learning technologist Library manager Collections manager
- 28. Evaluate: blended learning E-learningmanager »“I can begin to correlate outcomes for disabled student with online provision in different subject areas. Now I have proof that CPD in blended learning pays dividends for disabled students." 02/03/2017 Data and disabled students 28 Roll of roles… Senior manager Student union president Examination officer Study skills tutor Head of disability service Dyslexia specialist Assistive technologist Lecturer Learning technologist Library manager Collections manager E-learning manager
- 29. Evaluate: Data planning Dataanalyst »“I can begin to plan for the future in ways that can extend what is currently possible to do.With my colleagues I can begin to shape our data to really meet the needs of a wider group of learners.” 02/03/2017 Data and disabled students 29 Roll of roles… Senior manager Student union president Examination officer Study skills tutor Head of disability service Dyslexia specialist Assistive technologist Lecturer Learning technologist Library manager Collections manager E-learning manager Data analyst
- 30. Where this fitsin your institution http://bit.ly/2lFgpwv 02/03/2017 Data and disabled students 30
- 31. What Can AnalyticsContribute to Accessibility in e-Learning Systems and to Disabled Student’s Learning Martyn Cooper, Rebecca Ferguson and Annika Wolff
- 32. Context to theresearch • OU – distance educator • Larger than average no. of disabled students • Greater challenges in responding to individual needs of disabled learners at a distance • Students can declare a disability. But don’t necessarily know what type and no two disabled students are the same, anyway
- 33. The question • Canlearning analytics be used to identify modules with accessibility deficits?
- 34. First pass • Lookat average completion rates. – 1338 modules analysed – Can show 50% completion rate if 1 of 2 students with declared disability drops out. – Low numbers can skew results – Solution: analyse only modules with >25 disabled students = 668 modules
- 35. Refined approach usingodds ratios • Odds ratios can determine for 2 groups whether one group is more or less likely to achieve an outcome than another group. • It is a relative measure of the odds of one outcome occurring, given a particular criteria compared to odds of it happening in absence of the criteria • In this case: – Outcome is success of students on the course
- 36. Using odds ratiosto find accessibility issues • A bigger odds ratio = bigger disparity between groups • But - need to find threshold above which you can say there is a problem Threshold of > 3 looks sufficient to identify where accessibility is most likely factor to explain difference
- 37. Summary • Low numbersmake applying statistical measures very difficult • Not suitable for a large number of modules • Identifies where there might be a problem – but not how to fix it
- 38. Possible Future work •Use research to find ‘critical learning paths’ to identify accessibility issues on individual modules.
- 39. What next »Follow upemail with › feedback form › PDF of slides/notes › Links to Google form,Tricider votes, Niall and Julia blog posts »Link to recording (if we remembered to press Record!) »New blog post summarising issues and questions arising from session. 02/03/2017 Data and disabled students 39
- 40. jisc.ac.uk One CastleparkTower HillBristol BS2 0JA customerservices@jisc.ac.uk T 020 3697 5800 Thank you for listening Subject specialists Accessibility & Inclusion Julia.Taylor@jisc.ac.uk Alistair.McNaught@ jisc.ac.uk 02/03/2017 Data and disabled students 40 Margaret.McKay@ jisc.ac.uk
Editor's Notes
- #2 “Enshrined in the Jisc Code of practice is the principle that learning analytics should be for the benefit of all students“
- #3 Alistair Holding slide – while you wait
- #4 It is night. I am standing alone in 25 acres of scrubland in rural Nicaragua looking at a the Seven Sisters in a sky brimming with stars. A man with a rifle is walking towards me, demanding to know my business. I speak the Spanish of a 4 year old. The gun is pointing towards me. I persuade him that I am part of the project team he is supposed to be guarding rather than shooting. He asks why I am not with the rest of the team in the hut. Why am I out with the crickets, the dust and the snakes? I explain I love stars and planets. He tells me there was a news report that day about an asteroid that might hit earth in the 25th century. And what is the difference between an asteroid and a comet? The chirp of the crickets grows louder as I search my Beginners Spanish vocabulary for the chapter on the formation of the solar system. I pick up a stick and – by the light of his torch - draw the sun in the dust. I have a brainwave. Use complicated scientific words – they’re nearly all derived from Latin. It works. He is genuinely engaged. That night I dream I am talking to him and I see his meanings falling through the air. I have a net of language but the weave is wide and many of the meanings slip through without me catching them. I know that as I learn Spanish the weave of the net will become smaller and smaller and the most delicate of meanings will be caught. Then I will more fully know the people to whom I speak.
- #5 Alistair That net of language that catches finer and finer meanings from a speaker’s voice is exactly like the net of data that catches meaning from an institution’s processes, a tutor’s teaching, a library’s content or a learner’s progress. In this session we will look at how we can use the net of meanings to shape practice and personalise support. Just like there are ethical implications of our communication, our speaking and listening, there are also ethical implications for our net of data. And that’s where we start this session.
- #6 Alistair …have made you aware of the: ethical issues surrounding learning analytics and their resonance with disabled students opportunities and issues around disability definitions range of potential positive scenarios within institutions considerations relevant to your own institutional ambitions
- #7 In universities, colleges and schools we have lots of data, but it has traditionally been stored on paper and in filing cabinets – it’s been hard to access it and to analyse it. But this is changing. Many institutions are now collecting data more consistently and storing it in ways that it can be accessed easily and merged with other datasets to obtain greater insight on the effectiveness of educational processes. “learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” SoLAR – Society for Learning Analytics Research
- #8 Jisc’s learning analytics project consists of three core strands: a learning analytics service, a toolkit to help institutions develop their capacities, and a community for sharing experience across institutions.
- #9 Jisc has produced a code of practice for learning analytics, built from an extensive literature review. Questions which informed the code include: If a student is allowed to opt out of data collection and analysis could this have a negative impact on their academic progress? What should a student do if the suggestions are in conflict with their study goals? How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances?
- #10 The code of practice is available at http://jisc.ac.uk/guides/code-of-practice-for-learning-analytics
- #11 A screenshot of Tribal Student Insight, one of the products integrated with Jisc’s learning analytics architecture. It consists of a dashboard showing a student’s predicted risk scores for the modules she is taking.
- #12 Screenshots from Study Goal, the Jisc student app for learning analytics, showing an activity feed, course engagement and attainment data and a graph showing how your engagement compares with one of your friends over time.
- #13 Accessibility Considerations for Learning Analytics Remember that learning analytics is not assessment Avoid the labelling of individuals and reinforcing of prejudice and stereotypes Maintain disabled students’ confidentiality Handle the inference of disabilities from the analytics appropriately Ensure that the analytics do not unfairly single out disabled students Use analytics to identify modules where there appear to be accessibility issues Ensure that student-facing analytics are accessible Ensure that interventions are worded appropriately https://analytics.jiscinvolve.org/wp/2016/12/14/accessibility-considerations-for-learning-analytics/
- #14 Contacts Paul Bailey paul.bailey@jisc.ac.uk Niall Sclater niall.sclater@jisc.ac.uk Further Information: http://www.analytics.jiscinvolve.org Join: analytics@jiscmail.ac.uk
- #15 Straw poll
- #16 University of Wolverhampton identified >3o different sources of support for students. How can learning analytics support consistency and clarity? If you have different definitions of ‘disability data’ you end up telling different stories. What data do you collect? Does it have implicit accessibility data (e.g. from bursary, May not be getting the data due to funding changes (DSA) BUT the data would still be useful – How will you get students to continue to disclose? There is a need for evidence of impact (OFFA) Learning Analytics has great potential to shape provision Are learners needs reflected in the data you collect, how could this be improved?
- #17 Brainstorm scenarios Data 'stages' to consider: Deferent roles will play different parts but each will inform the other. Design of learner data What is collected (ethics, level of detail e.g. disability and other co-morbid factors, process and outcome), Design of interface (usability and accessibility), Support learner progress Documenting/disclosing barriers provides an additional method of early identification for support by tracking progress, Informing course design and learner attainment Improving learning and teaching practice Comparing progress of disabled versus non disabled learners Evaluate institutional/support services Usage of institution wide assistive technology (e.g. text to speech) Library uptake of productivity software, ebook usage. Etc.
- #18 Like to expand on this further - and think about prospective scenarios within a typical institution. In addition … and in true Academy Awards style lets think about the roll of rolls – what key people might be involved or indeed be responsible. Design stage and in particular Stakeholder engagement “We've invited a range of stakeholders to be involved in our learning analytics steering group - including support staff and people with accessibility needs” ” Senior managers might typically be involved in making decisions to initiate this
- #19 Design : Learner engagement “Students are actively involved in deciding what information they see on their own personal dashboard.” (Student Union President might typically be involed ) Here we’re considering the impact of learner advocacy - nothing about us without us
- #20 Design stage - Exam arrangements “Some people need access arrangements for exams. It’s always involved lots of consultation and meetings. Now its just a touch of a button” (Examination Officer)
- #21 Support – thinking about Consistency of support Now there’s less room for students to slip through the net because all the support services have the same information at the same time so can work together. (Study skills support tutor)
- #22 Support opportunities: responsiveness. Student data helps me track the progress of students who have disclosed a barrier to learning so we can respond more swiftly. (Head of disability services)
- #23 Support opportunities: Prioritise – don’t chase students who don’t need it! I used to spend ages chasing students who missed their appointments. Now I can instantly check their other progress and leave them alone if they’re succeeding. (Dyslexia specialist)
- #24 Evaluate: Support strategies As well as capturing information about a learners disability we've tried to find out more specific detail about the individual nature of a learners' need (e.g. keyboard only user, uses screen reading/magnification/text to speech software, has mental health issues) to help evaluate and audit our course demands and learner expectations .
- #25 Evaluate: Teaching approaches I can see how changes to my resources and activities have impacted on everyone's engagement, and particularly benefited my disabled students. (Lecturer)
- #26 Evaluate: Teaching approaches “I can see who is using the learning platform and how often . This makes it easy to see where content might be difficult to access.”
- #27 Evaluate: Library technology support We can now monitor the usage/uptake of enabling technology software in our library. This helps us to adopt a more targeted strategy for promotion of productivity tools to enhance the support we offer. (Library manager) Learner data is used to adapt services and where identified to change/improve support services. e.g. access to information on study skills tools, different ways to introduce library services, additional CPD for students and staff on enabling technology approaches, engagement with IT staff (ref Loughbourough Uni Pedestal for Progression project) In your organisation how likely is this practice to occur: answer on the poll. A: Already widespread practice B: Already well developed pockets C: Likely to be a willingness D Likely to be resistance
- #28 Evaluate: E-books and journals: ““I can see who is using the resources such as e-books and e-journals. If there are any anomalies I can ask why. By exploring use by different categories of student we can plan more effective intervention and support” Collections manager
- #29 Evaluate: blended learning “I can begin to correlate outcomes for disabled student with online provision in different subject areas. Now I have proof that CPD in blended learning pays dividends for disabled students.“E-learning manager
- #30 Evaluate: Data planning “I can begin to plan for the future in ways that can extend what is currently possible to do. Witj my colleagues I can begin to shape our data to really meet the needs of a wider group of learners.” Data analyst
- #31 5 minute activity http://bit.ly/2lFgpwv or http://www.tricider.com/brainstorming/2egTN8IYdK7 We are now like to invite you to reflect on some more scenarios in the context of your own organisation and your own role. I’ll add a link to this poll in the chat pane - http://bit.ly/2lFgpwv and we’d like to invite you to visit this page now – we’ll take 5 minutes to: 1.Review the 8 scenarios on the webpage 2.For each of these statements we’d like to invite you to vote for the top 3 that you think are most important for you, that you feel you would prioritise or indeed be the most helpful or achievable. 3.If you have time to - or you wish to – you might feel compelled to add a comment - please do add your own ‘argument’ or viewpoint under the pros and cons section 4.You may alternatively feel that you want to add an additional scenario – one that hasn’t been mentioned here. Lets take some time and come back in 5 minutes – This poll will be open for 100 plus days which means that you’ll be able to review what others have added, to reflect on these priorities in the context of your own organisation, share with colleagues and use as We’ve looked at the art of the possible, and now we are going to take the opportunity to hear about an example of practice learn about how one organisation has implemented approaches to use learner data to support disabled students.
- #32 What’s possible - Scenarios that make good use of learner information What Can Analytics Contribute to Accessibility in e-Learning Systems and to Disabled Student’s Learning Martyn Cooper, Rebecca Ferguson and Annika Wolff
- #33 Context to the research OU – distance educator Larger than average no. of disabled students Greater challenges in responding to individual needs of disabled learners at a distance Students can declare a disability. But don’t necessarily know what type and no two disabled students are the same, anyway
- #34 The question Can learning analytics be used to identify modules with accessibility deficits?
- #35 First pass Look at average completion rates. 1338 modules analysed Can show 50% completion rate if 1 of 2 students with declared disability drops out. Low numbers can skew results Solution: analyse only modules with >25 disbaled students = 668 modules
- #36 Refined approach using odds ratios Odds ratios can determine for 2 groups whether one group is more or less likely to achieve an outcome than another group. It is a relative measure of the odds of one outcome occurring, given a particular criteria compared to odds of it happening in absence of the criteria In this case: Outcome is success of students on the course
- #37 Using odds ratios to find accessibility issues A bigger odds ratio = bigger disparity between groups But - need to find threshold above which you can say there is a problem Threshold of > 3 looks sufficient to identify where accessibility is most likely factor to explain difference
- #38 Summary Low numbers make applying statistical measures very difficult Not suitable for a large number of modules Identifies where there might be a problem – but not how to fix it
- #39 Possible Future work Use research to find ‘critical learning paths’ to identify accessibility issues on individual modules.
- #40 What next? Follow up email with feedback form PDF of slides/notes Links to Google form, Tricider votes, Niall and Julia blog posts Link to recording (if we remembered to press Record!) New blog post summarising issues and questions arising from session.
- #41 Thank you for listening Subject specialists Accessibility & Inclusion Margaret.McKay@ jisc.ac.uk Julia.Taylor@jisc.ac.uk Alistair.McNaught@ jisc.ac.uk