Learning Analytics and Accessibility – what
can be done and pragmatic considerations
Martyn Cooper (IET)
CALRG Conference ...
Introduction
• Work is part of the eSTEeM project LA4DS-STEM
• Learning Analytics (LA) requires “Big Data”
• Particular in...
The Central Hypothesis
• In modules where the completion rate and the pass rate
are significantly lower for disabled stude...
The Data Set
• The “big data”:
–All Science and MCT Modules from presentation
2009B to and including 2013J (5 years)
–1452...
Odds Ratios Explanation
• Need a statistical useful comparison
–Using odds ratios [J.T.E. Richardson personal communicatio...
Odds Ratios Explanation cont.
• An odds ratio of 1 means that there is no difference in
the odds of the two groups’ member...
Completion % All Modules
-60.0
-40.0
-20.0
0.0
20.0
40.0
60.0
80.0
100.0
120.0
1
32
63
94
125
156
187
218
249
280
311
342
...
Completion % - Modules with
> 25 disabled students registered
-30.0
-20.0
-10.0
0.0
10.0
20.0
30.0
40.0
1
15
29
43
57
71
8...
Odds Ratios Completion
All Modules in Data Set
0.000
2.000
4.000
6.000
8.000
10.000
12.000
14.000
16.000
18.000
20.000
1
3...
Thresholds for Decisions
• The learning analytics needs to lead to a decision about
which modules include significant acce...
Future Work
• Liaison with Science Accessibility Specialist particularly
with reference to S104 and planning for the new L...
Discussion Points
• Learning Analytics approaches seem to be able to
identify major accessibility issues in modules
– Howe...
Questions
Institute of Educational Technology
The Open University
Walton Hall
Milton Keynes
MK7 6AA
www.open.ac.uk
Upcoming SlideShare
Loading in …5
×

Learning analytics and accessibility – #calrg 2015

677 views

Published on

Presentation at the Open University's Computers and Learning Research Group (CALRG) Conference 2015 on Learning Analytics and Accessibility - detecting accessibility deficits with Learning Analytics approaches

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
677
On SlideShare
0
From Embeds
0
Number of Embeds
74
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Odds ratios enable you to make comparisons across different module presentations when the underlying phenomenon (i.e. the completion rate) is varying.
  • Learning analytics and accessibility – #calrg 2015

    1. 1. Learning Analytics and Accessibility – what can be done and pragmatic considerations Martyn Cooper (IET) CALRG Conference 2015
    2. 2. Introduction • Work is part of the eSTEeM project LA4DS-STEM • Learning Analytics (LA) requires “Big Data” • Particular interest in retention and pass rates • Research Questions: • What can LA approaches tell us about the accessibility (to disabled students) of modules? • If they tell us anything is the approach useful?
    3. 3. The Central Hypothesis • In modules where the completion rate and the pass rate are significantly lower for disabled students than for nondisabled students then this is indicative of accessibility challenges in that module Questions: • What is significance indicated by in this case? • How confident are we that we are not measuring other factors that impact on performance such as: motivation; family circumstances; ability; educational background; etc.?
    4. 4. The Data Set • The “big data”: –All Science and MCT Modules from presentation 2009B to and including 2013J (5 years) –1452 presentations in total –% Completion and % Pass Rates –Disabled students vs Non-disabled students –Data set needed some “cleaning-up” before analysis Disabled No Yes Module Presentation Total No. % complete % pass No. % complete % pass A 2009B 1827 1605 63.4 61.6 222 56.3 52.7 B 2009J 2609 2282 67.4 66.3 327 59.0 56.9 C 2010B 1662 1492 61.5 60.1 170 60.6 58.2 … … … … … … … … …
    5. 5. Odds Ratios Explanation • Need a statistical useful comparison –Using odds ratios [J.T.E. Richardson personal communication] • If the probability of the members of Group 1 exhibiting a particular outcome is p then the odds of this are p/(1 − p) • If the probability of the members of Group 2 exhibiting that outcome is q, then the odds of this are q/(1 − q) • The odds ratio is the ratio between these odds (i.e. [p/(1 − p)]/[q/(1 − q)], which equals [p(1 − q)]/[q(1 − p)]) • Odds ratios vary from 0 (when p = 0 or q = 1) to infinity (when p = 1 or q = 0)
    6. 6. Odds Ratios Explanation cont. • An odds ratio of 1 means that there is no difference in the odds of the two groups’ members exhibiting the outcome (when p = q) • An odds ratio less than 1 means that the members of Group 1 are less likely to exhibit the outcome than are the members of Group 2; and an odds ratio greater than 1 means that the members of Group 1 are more likely to exhibit the outcome than are the members of Group 2 • N.B. - Whether an odds ratio is significantly different from 1 depends on the odds ratio itself and on the number of members in each group
    7. 7. Completion % All Modules -60.0 -40.0 -20.0 0.0 20.0 40.0 60.0 80.0 100.0 120.0 1 32 63 94 125 156 187 218 249 280 311 342 373 404 435 466 497 528 559 590 621 652 683 714 745 776 807 838 869 900 931 962 993 1024 1055 1086 1117 1148 1179 1210 1241 1272 1303 1334 1365 1396 1427 Disabled-Nondisabled % Complete Disabled-Nondisabled % Complete Cases % difference > 35% when low number disabled students registered
    8. 8. Completion % - Modules with > 25 disabled students registered -30.0 -20.0 -10.0 0.0 10.0 20.0 30.0 40.0 1 15 29 43 57 71 85 99 113 127 141 155 169 183 197 211 225 239 253 267 281 295 309 323 337 351 365 379 393 407 421 435 449 463 477 491 505 519 533 547 561 575 589 603 617 631 645 659 Nondisabled-Disabled % Complete Nondisabled-Disabled % Complete
    9. 9. Odds Ratios Completion All Modules in Data Set 0.000 2.000 4.000 6.000 8.000 10.000 12.000 14.000 16.000 18.000 20.000 1 36 71 106 141 176 211 246 281 316 351 386 421 456 491 526 561 596 631 666 701 736 771 806 841 876 911 946 981 1016 1051 1086 1121 1156 1191 1226 1261 1296 1331 Odds Ratios Complete (>1 if non-disabled outperform disabled) Odds Ratios Complete (>1 if non-disabled outperform disabled) All cases Odds Ratio >10 occur when low number of disabled students registered
    10. 10. Thresholds for Decisions • The learning analytics needs to lead to a decision about which modules include significant accessibility barriers for remedial action – The LA can tell you where a problem might be not what it is • Thresholds for decision making are arbitrary but informed by the data – A reasonable threshold for identifying accessibility problems seems to be an odds ratio of 4.0 or more
    11. 11. Future Work • Liaison with Science Accessibility Specialist particularly with reference to S104 and planning for the new Level 1 gateway modules • Liaison with the Science and MCT Data Wranglers to look for any correlation with the their data • Focus group(s) with MCT and Science staff of mock-ups of Learning Analytics Dashboards • Paper for LAK16 comparing with qualitative data from end of module surveys
    12. 12. Discussion Points • Learning Analytics approaches seem to be able to identify major accessibility issues in modules – However this needs testing and only possible by a detailed accessibility assessment of the module’s media and activities (this work not currently funded) • LA approaches only valid on modules with a significant number of disabled students – suggest a minimum of 25 • Even with 25 disabled students per module really need to evaluate over multiple presentations to identify issues – Does this mean the approach is less useful than responding to student complaints, or proper accessibility evaluation in production, etc.?
    13. 13. Questions
    14. 14. Institute of Educational Technology The Open University Walton Hall Milton Keynes MK7 6AA www.open.ac.uk

    ×