Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Can computer-marked final assessment improve retention?

699 views

Published on

Distance learning modules (particularly low-cost introductory and enrichment modules) may show poor retention compared to traditional campus courses. The perceived difficulty of exams and end-of-module assessments (EMA) appears to deter some students from submitting. In contrast, interactive computer-marked assignments (iCMA) are typically attempted by most students.
Can retention therefore be improved by changing the format of part of the final assessment to an iCMA?
Robotics and the meaning of life is a 10-point, 10-week general-interest Open University module. The assessment comprised a mid-module iCMA and a final written EMA. The iCMA (a Moodle quiz) provided detailed feedback only after the submission deadline. The EMA included short-answer questions, a programming question and an essay. The EMA was script-marked and feedback limited to overall score and performance profile provided well after the end of the course.
The intervention simply replaced the script-marked short-answer questions by a second iCMA covering the same content with similar questions. The programming and essay questions were retained unchanged as a written, script-marked EMA.
The hypothesis to be tested was that retention would increase: students would be more likely to submit the final iCMA, their confidence would increase, and they would be motivated to submit the written EMA.
Quantitative data were gathered for patterns of submission, course completion and pass rates for two presentations (124 and 220 students); data were also available for thirteen previous presentations (1814 students). Structured interviews were carried out to probe student preferences, confidence and engagement.
More students submitted the iCMA (86%) than the EMA (81%). Although they had the same deadline, 91% of students submitted the iCMA before the EMA. They submitted the iCMA well in advance of the deadline (median 4 days 15 hrs) but kept the EMA open as long as possible (median 18 hrs before deadline; 11% submitted in the final hour). These patterns strongly suggest that students were more confident with the iCMA than the EMA. Completion rates were the highest recorded: 88% and 89% compared to 79% for pre-intervention presentations. Overall pass rates were also improved (83% and 85% c.f. 76%). This can be ascribed to improved submission rates alone: the pass rate and mean scores among those who submit were unchanged giving confidence that the assessment difficulty was unaltered.
Student interviews suggested that students did attempt the final iCMA before the EMA and had greater confidence in obtaining a good mark for the iCMA than the EMA. Students valued the mix of assessment methods and felt it produced a robust result; although some expressed concern over the correctness of computer marking, they appreciated the detailed feedback it provided.
This intervention suggests that a change of assessment format can improve student engagement and pass rates without compromising rigour.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Can computer-marked final assessment improve retention?

  1. 1. Can computer-marked finalassessment improve retention?Jon RosewellDept of Communications and Systems,Faculty of Maths, Computing and Technology,The Open University, UKALT-C, 11-13th Sept 2012
  2. 2. Robotics and the meaning of life:a practical guide to things that think (T184)• Open University ‘taster’ distance learning course• 10-point, 10-week, 100 hours• 2003 – 2011, 2387 students• Components: – Website – OU-RobotLab practical – software with LEGO kit or simulator – Isaac Asimov: I, Robot – James May on DVD – Moderated, peer-support forums
  3. 3. Challenge of retention• ‘Taster’ distance learning modules, not core to degree• Mid-course CMA (Moodle quiz) & final EMA (script-marked)• Students enjoyed and completed course …but daunted by EMA• Students who submitted nearly all passed – too late!• Not submitting EMA  not completing module  poor retention statistic  setting up students to failRetention = module completion (not pass)Progression = continue to study another OU module
  4. 4. Can a final CMA improve retention?• iCMA encourages engagement and motivation• Submission of CMA generally high (T184: 87%, n=2158)• So add final CMA(CME) and reduce EMA• iCMA / iCME are Moodle quizzes – Open at start of course; single submission by traditional cut-off date – Questions can be revisited – Question level feedback immediately after cut-off date – Even without adaptive feedback, learner can judge their own performance • Builds confidence • Lowers perceived barrier to pass
  5. 5. Assessment strategyOriginal New• 10% – mid-course iCMA • 10% – mid-course iCMA• 90% – final written EMA – Part I – short answer qns • 30% – final iCME – Part II – prog & essay qns • 60% – final written EMA – prog & essay qns
  6. 6. Hypothesis:• Students will attempt iCME/quiz first, before written EMA  They will thus complete the module• This will improve their confidence• They will then be motivated to attempt EMA They will thus pass the module
  7. 7. Did more submit iCME? EMA Tot al Not Submitt submitte ed d CM Not 44 1 45 13 E submitte % d Submitted T184 2011E & 2011J presentations 19 281 300 87 %
  8. 8. Did students submit iCME first?CME clearly first 214 76%EMA clearly first 9 3%CME & EMA together (<1hr) 58 21% CME slightly earlier 43 15% EMA slightly earlier 15 5%Total 281 100% T184 2011E & 2011J presentation
  9. 9. Leaving it to the last minute: submitting EMA200180 Median: 18hrs before deadline160 60% in final 24hrs140 11% in final hour120100 EMA submitted 80 60 40 20 0 12-Nov 05-Nov 19-Nov 26-Nov 01-Oct 08-Oct 15-Oct 22-Oct 29-Oct 03-Dec 10-Dec T184 2011J
  10. 10. Not leaving it to the last minute: submitting iCME200 Median: 4 days 15 hrs150 iCME started100 iCME completed50 0 05-Nov 12-Nov 19-Nov 26-Nov 08-Oct 01-Oct 15-Oct 22-Oct 29-Oct 03-Dec 10-Dec T184 2011J
  11. 11. T184 completion rates100%95%90%85% May80% Oct75%70% Introduction of65% CME60% 2004 2005 2006 2007 2008 2009 2010 2011
  12. 12. T184 completion rates100%95%90%85% May80% Oct75%70% Introduction of65% CME60% 2004 2005 2006 2007 2008 2009 2010 2011
  13. 13. T184 completion rates100%95%90%85% May80% Oct75%70% Introduction of65% CME60% 2004 2005 2006 2007 2008 2009 2010 2011
  14. 14. T184 pass rates100%95%90%85% May80% Oct75%70% Introduction of65% CME60% 2004 2005 2006 2007 2008 2009 2010 2011
  15. 15. Interviews with students I thought the balance of assessment was about the best I’d come across. Interesting computer-based tests … plus the programming elements … an interesting [EMA] with a choice of two essays. … The assessment contributed to the enjoyment of the whole course. By the time I’d finished the assessment I’d felt I’d learnt more. … I also felt fairly confident that I’d got enough marks on the computer based test.
  16. 16. Interviews with students [Early on] get an achievement and get a bit of success, … it was really easy that first CMA. … So an early success and then you know at the end you’ve got some short answers in the CMA that you think pick up your score a bit, and then you’ve got the EMA where you can do even better. I think it’s spot on.
  17. 17. Interviews with students … as long as they feel it’s rigorous … it didn’t have that kind of ITV quiz thing, you know ‘what’s the capital of England’ or something. … There was a combination with ‘you’ve done this bit now, and now you need to tell us about it, now you need to describe it, now you need to do some programming and show us your work’. It felt like there was real rigour to it so it didn’t feel like ‘this is a quick and dirty kind of interactive quiz’.
  18. 18. Interviews with studentsStudent: I think it was excellent. … The only thing I don’t like is having to wait till September for my result!Me: So that’s another reason for introducing that computer marked exam … it gives students feedback immediately rather than a long delay.Student: Yes, so you know where you’ve gone wrong. That’s really good, and it cheers you up as well, not an idiot after all!
  19. 19. Interviews with students “I enjoyed all of it really. I thought the course materials were brilliantly laid out, I thought it was very clear, concise. It gave you enough about the subject to make you interested and wanted you to do more.” “As I say it was a pleasure doing it. I hope I’ve passed, but I just think it was a joy doing it, it was really fun, you learnt something, it really complemented what I learnt last year and it’s just made me keen to do other stuff, or try and find out more about it.”
  20. 20. Thanks for listening…Jon RosewellJ.P.Rosewell@open.ac.uk
  21. 21. The end … of T184 Block 3 birth … of TM129
  22. 22. A tale of several short modules Based on Relevant Knowledge, 10 point, 10 week, level one modules with mid-course CMA and final EMA• A module with poor completion• A module that satisfies student needs• A module that is a paragon of good practiceRetention = module completion (not pass)Progression = continue to study another OU module
  23. 23. A module with poor completion• The mid-module CMA is low-stakes and easy to submit – ~80% students submit mid-course CMA• The final EMA is a project, submitted via eTMA system – Only 50% students submit final EMA• Many new students do not continue to study• Very few students subsequently complete a qualification
  24. 24. A ‘one-off’ module• Most new students want this module, not a qualification• Most new students won’t register for a further module• Many students therefore see no need to stress over EMA – Only 50% of students submit EMA• Most students complete the course material – ~80% complete CMA – Engaged on forums
  25. 25. Can you guess what it is yet?• A module with poor completion – T180 Living with the net• A module that satisfies student needs – T180 Living with the net
  26. 26. Does ‘one-off’  poor retention?An ‘M-world’ module• T180 Living with the Net (2004-2006) as an example• Some studying as ‘one-off’• Students enjoyed and completed course but daunted by EMA• Students who submitted nearly all passed – too late!• Not submitting EMA  not completing module  poor retention statistic (51%)  HEFCE penalty• Setting up students to fail?
  27. 27. Can an iCME improve retention?• Moodle quiz – Same format as mid-course iCMA – Open at start of course; single submission by traditional cut-off date – Questions can be revisited – Question level feedback immediately after cut-off date (cf OU anodyne performance profile 3 months later…)• Even without adaptive feedback, learner can judge their own performance – Builds confidence – Lowers barrier to pass: relatively low score on EMA programming and essay questions sufficient to pass

×