Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Student Data: Data is knowledge
– putting the knowledge back in the
students’ hands
Owen Corrigan
Mark Glynn
Aisling McKen...
Outline
• Motivation and goals
• Selecting the modules
• Study by numbers
• The interventions
- What the student sees
• Wh...
Motivation
Study by numbers
• 17 Modules across the
University (first year, high
failure rate, use Loop,
periodicity, stability of
co...
Total Moodle Activity – notice the
periodicity
One example module – ideal !
Modules which work well …
• Have periodicity (repeatability) in Moodle access
• Confidence of predictor increases over tim...
No significant difference in the entry profiles of
participants vs. non-participants overall
PredictEd Participant Profile
LG116 – Predictor confidence (ROC AUC)
SS103
What did the students say?
Students who took part were asked to complete a short
survey at the start of Semester 2 - N=133...
33% said they changed how they
used Loop. We asked them how?
• Studied more
– “More study”
– “Read some other articles onl...
Did you change Loop usage for
other modules?
• Most who commented used Loop more often for other modules
– “More often”
– ...
Subject Description Non-Participant Participant
BE101 Introduction to Cell Biology and Biochemistry 58.89 62.05
CA103 Comp...
Questions and discussion…
Contact details
• mark.glynn@dcu.ie
• @glynnmark
• http://enhancingteaching.com
• https://predictedanalytics.wordpress.com/
Additional slides
Student Interventions:
Feedback
The Interventions – Lecturers’ Experience
LG116: Introduction to Politics
Students / year = ~110
Pass rate = 0.78
Importance of Ethics
• Ethics are important to ensure safety of participants and
researchers
• Educational Data Analytics ...
So much student data we could
useDemographics
• Age, home/term address, commuting distance, socio-economic status, family
...
Building classifiers for each
week/each module
Training Data
Testing
Notes on model confidence
• Y axis is confidence in AUC ROC (not probability)
• X axis is time in weeks
• 0.5 or below is ...
0%
20%
40%
60%
80%
100%
LG116 MS136 LG101 HR101 LG127 ES125 BE101 SS103 CA103 CA168
Workshops
Wikis
Forums
Assignments
Qui...
BE101: Intro to Cell Biology
Results / year = ~300
Pass rate = 0.86
BE101
SS103: Physiology for Health Sciences
Results / Year = ~150
Pass rate = 0.92
MS136
LG101
HR101
CA103
Some unusable modules
Modules where the ROC AUC increases slowly
(e.g stays below 0.6) e.g. PS122
Timescale for Rollout
• Still some issues on Moodle access log data transfer
to be resolved
• Still have to resolve studen...
Why did you take part?
• The majority of students
wanted to learn/monitor
their performance
• Many others were curious
• S...
How easy was it to understand the
information in the emails ?
(1= not at all easy, 5 = extremely easy)
• Average 3.97 (SD=...
Week 3
Training Data
Testing
Week 4
Training Data
Testing
Week 5
Training Data
Testing
Week 6
Training Data
Testing
Week 7
Training Data
Testing
Week 8
Training Data
Testing
Week 9
Training Data
Testing
Upcoming SlideShare
Loading in …5
×

Predicted project Hatfield UK 2015 M Glynn

3,095 views

Published on

A copy of the slides produced to highlight the Predicted project that is mining data from our VLE and using it to predict academic success for students

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Predicted project Hatfield UK 2015 M Glynn

  1. 1. Student Data: Data is knowledge – putting the knowledge back in the students’ hands Owen Corrigan Mark Glynn Aisling McKenna Alan F. Smeaton Sinéad Smyth @glynnmark
  2. 2. Outline • Motivation and goals • Selecting the modules • Study by numbers • The interventions - What the student sees • What the students said • The results
  3. 3. Motivation
  4. 4. Study by numbers • 17 Modules across the University (first year, high failure rate, use Loop, periodicity, stability of content, Lecturer on-board) • Offered to students who opt-in or opt-out, over 18s only • 76% of students opted-in, 377 opted-out, no difference among cohorts • 10,245 emails sent to 1,184 students who opted-in over 13 weekly email alerts
  5. 5. Total Moodle Activity – notice the periodicity
  6. 6. One example module – ideal !
  7. 7. Modules which work well … • Have periodicity (repeatability) in Moodle access • Confidence of predictor increases over time • Don't have high pass rates (< 0.95) • Have large number of students, early-stage
  8. 8. No significant difference in the entry profiles of participants vs. non-participants overall PredictEd Participant Profile
  9. 9. LG116 – Predictor confidence (ROC AUC)
  10. 10. SS103
  11. 11. What did the students say? Students who took part were asked to complete a short survey at the start of Semester 2 - N=133 (11% response rate) Question Group 1 (more detailed email) Group 2 % of respondents who opted out of PredictED during the course of the semester 4.5% 4.5% % who changed their Loop usage as a result of the weekly emails 43.3% 28.9% % who would take part again/are offered and are taking part again 72.2% (45.6%/ 26.6% ) 76.6% (46% /30.6% )
  12. 12. 33% said they changed how they used Loop. We asked them how? • Studied more – “More study” – “Read some other articles online” – “Wrote more notes” – “I tried to apply myself much more, however yielded no results” – “It proved useful for getting tutorial work done” • Used Loop more – “I tried harder to engage with my modules on loop” – “I think as it is recorded I did not hesitate to go on loop. And loop as become my first support of study.” – “I logged on more” – “I read most of the extra files under each topic, I usually would just look at the lecture notes.” – “I looked at more of the links on the course nes pages, which helped me to further my understanding of the topics” – “I learnt how often I need to log on to stay caught up.”
  13. 13. Did you change Loop usage for other modules? • Most who commented used Loop more often for other modules – “More often” – “More efficient” – “Used loop more for other modules when i was logging onto loop for the module linked to PredictED” – “Felt more motivated to increase my Loop usage in general for all subjects” One realised that Lecturers could see their Loop activity “I realised that since teachers knew how much i was using loop, i had to try to mantain pages long on so it looked as if i used it a lot”
  14. 14. Subject Description Non-Participant Participant BE101 Introduction to Cell Biology and Biochemistry 58.89 62.05 CA103 Computer Systems 70.28 71.34 CA168 Digital World 63.81 65.26 ES125 Social&Personal Dev with Communication Skills 67.00 66.46 HR101 Psychology in Organisations 59.43 63.32 LG101 Introduction to Law 53.33 54.85 LG116 Introduction to Politics 45.68 44.85 LG127 Business Law 60.57 61.82 MS136 Mathematics for Economics and Business 60.78 69.35 SS103 Physiology for Health Sciences 55.27 57.03 Overall Dff in all modules 58.36 61.22 Average scores for participants are higher in 8 of the 10 modules analysed, significantly higher in BE101, and CA103. MS136 Module Average Performance Participants vs. Non-Participants
  15. 15. Questions and discussion…
  16. 16. Contact details • mark.glynn@dcu.ie • @glynnmark • http://enhancingteaching.com • https://predictedanalytics.wordpress.com/
  17. 17. Additional slides
  18. 18. Student Interventions: Feedback
  19. 19. The Interventions – Lecturers’ Experience
  20. 20. LG116: Introduction to Politics Students / year = ~110 Pass rate = 0.78
  21. 21. Importance of Ethics • Ethics are important to ensure safety of participants and researchers • Educational Data Analytics is a new area of research – Not much previous research to highlight possible ethical issues – Requires extensive ethical consideration • We have spent a lot of time this Summer preparing a DCU REC submission – We’ve submitted and had approval for a test case – We’ve met with REC chair to brief him • We are following the 8 Principles set out by the Open University who are at EXACTLY the same stage as us
  22. 22. So much student data we could useDemographics • Age, home/term address, commuting distance, socio-economic status, family composition, school attended, census information, home property value, sibling activities, census information Academic Performance • CAO and Leaving cert, University exams, course preferences, performance relative to peers in school Physical Behaviour • Library access, sports centre, clubs and societies, eduroam access yielding co- location with others and peer groupings, lecture/lab attendance, Online Behaviour • Mood and emotional analysis of Facebook, Twitter, Instagram activities, friends and their actual social network, access to VLE (Moodle)
  23. 23. Building classifiers for each week/each module Training Data Testing
  24. 24. Notes on model confidence • Y axis is confidence in AUC ROC (not probability) • X axis is time in weeks • 0.5 or below is a poor result • Most Modules start at 0.5 when we don't have much information • 0.6 is acceptable, 0.7 is really good (for this task) • The model should increase in confidence over time • Even if confidence overall increases, due to randomness the confidence may go up and down • It should trend upwards to be a valid model and viable module choice
  25. 25. 0% 20% 40% 60% 80% 100% LG116 MS136 LG101 HR101 LG127 ES125 BE101 SS103 CA103 CA168 Workshops Wikis Forums Assignments Quizzes scorm lesson choice feedback database glossary wiki url book pages folders files Course content
  26. 26. BE101: Intro to Cell Biology Results / year = ~300 Pass rate = 0.86
  27. 27. BE101
  28. 28. SS103: Physiology for Health Sciences Results / Year = ~150 Pass rate = 0.92
  29. 29. MS136
  30. 30. LG101
  31. 31. HR101
  32. 32. CA103
  33. 33. Some unusable modules Modules where the ROC AUC increases slowly (e.g stays below 0.6) e.g. PS122
  34. 34. Timescale for Rollout • Still some issues on Moodle access log data transfer to be resolved • Still have to resolve student name / email address / Moodle ID / student number • Still to resolve timing of when we can get new registration data, updates to registrations (late registrations, change of module, change of course, etc.) … • Should we get new, “clean” data each week ?
  35. 35. Why did you take part? • The majority of students wanted to learn/monitor their performance • Many others were curious • Some were interested in the Research aspect • Some were just following advice • Others were indifferent
  36. 36. How easy was it to understand the information in the emails ? (1= not at all easy, 5 = extremely easy) • Average 3.97 (SD= 1.07) • Very few had comments to make (19/133) – Most who commented wanted more detail.
  37. 37. Week 3 Training Data Testing
  38. 38. Week 4 Training Data Testing
  39. 39. Week 5 Training Data Testing
  40. 40. Week 6 Training Data Testing
  41. 41. Week 7 Training Data Testing
  42. 42. Week 8 Training Data Testing
  43. 43. Week 9 Training Data Testing

×