RIDE 2010 presentation - Using interactive computer-based assessment to support beginning distance learners of science


Published on

Research in Distance Education: impact on practice conference, 27 October 2010. Presentation in Design for Learning Strand by Sally Jordan, Open University Science Faculty.
More details at www.cde.london.ac.uk.

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

RIDE 2010 presentation - Using interactive computer-based assessment to support beginning distance learners of science

  1. 1. Using interactive computer-based assessment to support beginning distance learners of science Sally Jordan 27th October 2010 Centre for Distance Education Research in Distance Education
  2. 2. My plan • What we did • Why we did it • How we did it • Evaluation • How far is it appropriate to go?
  3. 3. What we did 1 • Interactive computer-marked assignments (iCMAs) as part of the assessment strategy of a module, alongside tutor-marked assignments; • iCMAs are summative but lightly weighted; • Their primary purpose is formative; • Other modules use iCMAs in a purely formative way, with students being able to repeat questions and try different variants of them for extra practice. • S104 website iCMA42
  4. 4. Why we did it 1 We use interactive computer marked assignments (iCMAs) alongside tutor-marked assignments (TMAs) to: • Provide instantaneous feedback – and an opportunity for students to act on that feedback; • Provide ‘little and often’ assessment opportunities and so help students to pace their studies; • Act as ‘a tutor at the student’s elbow’; • To help students, in conjunction with their tutor, to monitor their own progress. We use a range of question types, going beyond those where students select from pre-determined answers to those where they have to write in their own words.
  5. 5. What we did 2 • Diagnostic materials to help students find the correct starting point in their Open University study; • In the Science Faculty we initially used printed ‘Are you ready for?’ materials; • We are moving towards interactive diagnostic materials, e.g. ‘Are you ready for science study?’/’Are you ready for S104?’, together used by more than 3000 people every month. Study and the OU website AYRF S104
  6. 6. Why we did it 2 • The Open University operates an open entry policy to its undergraduate modules; • Getting students on the correct module in the first place is the most important factor in their subsequent retention, progression and success; • We want people to make decisions on the basis of their prior knowledge and skills, but these can be difficult to quantify; • The quizzes advise on the basis of what people can actually do rather than what they think they can do; • Making the quizzes online and interactive means that students have to engage with the questions rather than just looking at the answers.
  7. 7. How we did it • Most of our questions are written in OpenMark and sit within the Moodle virtual learning environment; • For short-answer free-text questions, we initially used answer-matching software provided by Intelligent Assessment Technologies (IAT), now we use PMatch (part of OpenMark); • Real student responses were used in developing the answer matching. • Questions were improved in the light of comments from students and tutors and in the light of data-analysis.
  8. 8. Evaluation Has included: • Human-computer marking comparison • Computer-computer marking comparisons; • Questionnaires and interviews; • Student observation (in usability laboratory); • Data-analysis : used to explore factors that impact on student engagement with e-assessment; • Further questionnaires and interviews.
  9. 9. Some conclusions • Students seem to like iCMAs: ∀ ∼90% of students agree with the statement ‘Answering iCMA questions helps me learn’. • Examples of comments from students: ‘iCMAs are a brilliant tool in building confidence’ ‘it’s more like having an online tutorial than taking a test’. • And from a tutor ‘a great example of how online assessment can aid learning.’ • But minor errors and ambiguities in individual questions can be off-putting to students, so it is important to monitor use.
  10. 10. Some conclusions cont. • Students appear to engage with summative iCMA questions at a deeper level than when they are in formative-only use; • However there are issues, especially preoccupation with the minutiae of grading; • Does the answer lie in thresholding or in formative-only use with a ‘carrot’?; • Exams are a stronger motivator – students decide for themselves to use iCMAs in revision.
  11. 11. How far is it appropriate to go? • I don’t see online assessment as a panacea; • Some learning outcomes are easier than others to assess in this way; • Free text questions require students to construct a response, but there still need to be definite ‘right’ and ‘wrong’ answers (though not necessarily a single right answer); • However, online assessment provides instantaneous feedback and has been shown to be more accurate than human markers. It can free up human markers for other tasks. It also has huge potential for diagnostic use.
  12. 12. Useful links PMatch demonstration https://students.open.ac.uk/openmark/omdemo.pm2009/ ‘Are you ready for S104?’ (diagnostic quiz, showing a range of question types) https://students.open.ac.uk/openmark/science.level1ayrf.s 104/ OpenMark examples site http://www.open.ac.uk/openmarkexamples/index.shtml COLMSCT iCMA initiative http://www.open.ac.uk/colmsct/projects/sallyjordan
  13. 13. For further information: • Jordan, S. & Mitchell, T. (2009) E-assessment for learning? The potential of short free-text questions with tailored feedback. British Journal of Educational Technology, 40, 2, 371-385. • Butcher, P.G. & Jordan, S.E. (2010) A comparison of human and computer marking of short free-text student responses. Computers & Education, 55, 489-499.
  14. 14. Sally Jordan Staff Tutor in Science The Open University in the East of England Cintra House 12 Hills Road Cambridge CB2 1PF s.e.jordan@open.ac.uk website: http://www.open.ac.uk/science/people/people- profile.php?staff_id=Sally%26%26Jordan blog: http://www.open.ac.uk/blogs/SallyJordan/