Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Moodle Quiz at the Open University: how we use it & how that helps students

4,100 views

Published on

My presentation at the 2015 Dublin Moodle Moot.

Published in: Education

The Moodle Quiz at the Open University: how we use it & how that helps students

  1. 1. The Moodle Quiz at the Open University: how we use it & how that helps students Tim Hunt, Information Technology MoodleMoot Dublin 2015
  2. 2. The Moodle Quiz at the Open University: how we use it & how that helps students Tim Hunt, Information Technology MoodleMoot Dublin 2015 many dedicated OU teaching staff
  3. 3. Alternate titles Lots of interesting graphs The scholarship of teaching and learning in practice
  4. 4. Overall use
  5. 5. It matters
  6. 6. End of module survey results 0% 20% 40% 60% 80% 100% KPI 01: Overall, I am satisfied with the quality of this module. Q33 Clear understanding of what was required to complete the assessed work Q34 the assessment activities supported my learning
  7. 7. It makes a difference
  8. 8. Level 3 physics (SM358)
  9. 9. SM358 assessment tasks 0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs 0 iCMAs 1 iCMA 2 iCMAs 3 iCMAs 4 iCMAs 5 iCMAs 6 iCMAs
  10. 10. Students completing tasks 0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs 0 iCMAs 11.6% 3.4% 1.5% 0.5% 0.5% 1 iCMA 1.5% 1.0% 2 iCMAs 1.5% 2.4% 1.5% 3 iCMAs 1.5% 4 iCMAs 5.3% 2.4% 5 iCMAs 0.5% 3.9% 5.8% 8.2% 6 iCMAs 0.5% 0.5% 5.8% 5.8% 34.3%
  11. 11. Average exam scores 0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs 0 iCMAs 6.0 1 iCMA 2 iCMAs 17.0 24.0 3 iCMAs 60.0 4 iCMAs 43.7 62.0 5 iCMAs 23.0 46.0 62.6 69.5 6 iCMAs 35.3 60.8 77.5
  12. 12. Exam scores vs prediction 0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs 0 iCMAs −20.8 1 iCMA 2 iCMAs −43.9 −27.5 3 iCMAs −9.0 4 iCMAs −15.6 +1.8 5 iCMAs −3.8 −11.1 +1.4 +2.4 6 iCMAs −17.1 +3.4 +4.6
  13. 13. Changing assessment type
  14. 14. T184 robotics & meaning of life Before 10% mid-course iCMA 90% final written EMA part 1 short-answer part 2 programming & essays After 10% mid-course iCMA 30% final iCMA 60% final written EMA
  15. 15. Module completion rates T184 completion rates 60% 65% 70% 75% 80% 85% 90% 95% 100% 2004 2005 2006 2007 2008 2009 2010 2011 May Oct Introduction of CME
  16. 16. T184 completion rates 60% 65% 70% 75% 80% 85% 90% 95% 100% 2004 2005 2006 2007 2008 2009 2010 2011 May Oct Introduction of CME Module completion rates
  17. 17. T184 completion rates 60% 65% 70% 75% 80% 85% 90% 95% 100% 2004 2005 2006 2007 2008 2009 2010 2011 May Oct Introduction of CME Module completion rates
  18. 18. Deadlines
  19. 19. SM358 iCMA51 submit date 2010 advisory deadline
  20. 20. SM358 iCMA51 submit date 2010 advisory deadline 2012 hard deadline
  21. 21. Grades
  22. 22. Optional quizzes
  23. 23. Compulsory quizzes
  24. 24. Do students read feedback?
  25. 25. Interactive with multiple tries
  26. 26. Same question in two quizzes
  27. 27. Can computers grade sentences?
  28. 28. Spoiler: Yes!
  29. 29. How good are humans? Question Number of responses in analysis Percentage of responses where the human markers were in agreement with question author Percentage of responses where computer marking was in agreement with question author Range for the 6 human markers Mean of the 6 human markers A 189 97.4 – 100. 98.9 99.5 B 248 83.9 – 97.2 91.9 97.6 C 150 80.7 – 94.0 86.9 94.7 D 129 91.5 – 98.4 96.7 97.6 E 92 92.4 – 97.8 95.1 98.9 F 129 86.0 – 97.7 90.8 97.7 G 132 66.7 – 90.2 83.2 89.4
  30. 30. Comparing three algorithms Question Number of responses in analysis Percentage of responses where computer marking was in agreement with question author Computational linguistics Algorithmic manipulation of keywords IAT Pattern-match Regular Expressions A 189 99.5 99.5 98.9 B 248 97.6 98.8 98.0 C 150 94.7 94.7 90.7 D 129 97.6 96.1 97.7 E 92 98.9 96.7 96.7 F 129 97.7 88.4 89.2 G 132 89.4 87.9 88.6
  31. 31. Summary
  32. 32. References • Overall iCMA usage numbers collated by Phil Butcher. • End of module server results from Student Analytics in the Institute of Educational Technology via Linda Price. • SM358 data from John Bolton • T184 data from Jon Rosewell • SDK125, S141, S151 & S240 data from Sally Jordan • Jordan, Sally (2014). Using e-assessment to learn about students and learning. International Journal of e-Assessment, 4(1) • Jordan, Sally (2014). Adult science learners’ mathematical mistakes: an analysis of responses to computer-marked questions. European Journal of Science and Mathematics Education, 2(2) pp. 63–86. • Jordan, Sally (2012). Short-answer e-assessment questions : five years on. In: 2012 International Computer Assisted Assessment Conference, 10-11 July 2012, Southampton. • Pattern-match question type for Moodle https://moodle.org/plugins/view/qtype_pmatch
  33. 33. Key points Getting the assessment right is important Online quizzes can be powerful learning tools Dig into the data and you can learn • Is this quiz working? • What are my students’ misconceptions? Computers can grade much more than multi-choice but only on behalf of a teacher Want more? I’m giving this talk again at #iMoot15

×