Research in Distance Education: impact on practice conference, 27 October 2010. Presentation in Design for Learning Strand by Sally Jordan, Open University Science Faculty. More details at www.cde.london.ac.uk.

1. Using interactive computer-based
assessment to support beginning
distance learners of science
Sally Jordan
27th
October 2010
Centre for Distance Education Research in Distance Education

2. My plan
• What we did
• Why we did it
• How we did it
• Evaluation
• How far is it appropriate to go?

3. What we did 1
• Interactive computer-marked assignments (iCMAs) as
part of the assessment strategy of a module, alongside
tutor-marked assignments;
• iCMAs are summative but lightly weighted;
• Their primary purpose is formative;
• Other modules use iCMAs in a purely formative way,
with students being able to repeat questions and try
different variants of them for extra practice.
• S104 website iCMA42

4. Why we did it 1
We use interactive computer marked assignments
(iCMAs) alongside tutor-marked assignments (TMAs) to:
• Provide instantaneous feedback – and an opportunity for
students to act on that feedback;
• Provide ‘little and often’ assessment opportunities and
so help students to pace their studies;
• Act as ‘a tutor at the student’s elbow’;
• To help students, in conjunction with their tutor, to
monitor their own progress.
We use a range of question types, going beyond those
where students select from pre-determined answers to
those where they have to write in their own words.

5. What we did 2
• Diagnostic materials to help students find the correct
starting point in their Open University study;
• In the Science Faculty we initially used printed ‘Are you
ready for?’ materials;
• We are moving towards interactive diagnostic materials,
e.g. ‘Are you ready for science study?’/’Are you ready for
S104?’, together used by more than 3000 people every
month.
Study and the OU website AYRF S104

7. Why we did it 2
• The Open University operates an open entry policy to its
undergraduate modules;
• Getting students on the correct module in the first place
is the most important factor in their subsequent
retention, progression and success;
• We want people to make decisions on the basis of their
prior knowledge and skills, but these can be difficult to
quantify;
• The quizzes advise on the basis of what people can
actually do rather than what they think they can do;
• Making the quizzes online and interactive means that
students have to engage with the questions rather than
just looking at the answers.

8. How we did it
• Most of our questions are written in OpenMark and sit
within the Moodle virtual learning environment;
• For short-answer free-text questions, we initially used
answer-matching software provided by Intelligent
Assessment Technologies (IAT), now we use PMatch
(part of OpenMark);
• Real student responses were used in developing the
answer matching.
• Questions were improved in the light of comments from
students and tutors and in the light of data-analysis.

9. Evaluation
Has included:
• Human-computer marking comparison
• Computer-computer marking comparisons;
• Questionnaires and interviews;
• Student observation (in usability laboratory);
• Data-analysis : used to explore factors that impact on
student engagement with e-assessment;
• Further questionnaires and interviews.

10. Some conclusions
• Students seem to like iCMAs:
∀ ∼90% of students agree with the statement ‘Answering
iCMA questions helps me learn’.
• Examples of comments from students:
‘iCMAs are a brilliant tool in building confidence’
‘it’s more like having an online tutorial than taking a test’.
• And from a tutor
‘a great example of how online assessment can aid
learning.’
• But minor errors and ambiguities in individual questions
can be off-putting to students, so it is important to
monitor use.

11. Some conclusions cont.
• Students appear to engage with summative iCMA
questions at a deeper level than when they are in
formative-only use;
• However there are issues, especially
preoccupation with the minutiae of grading;
• Does the answer lie in thresholding or in
formative-only use with a ‘carrot’?;
• Exams are a stronger motivator – students
decide for themselves to use iCMAs in revision.

12. How far is it appropriate to go?
• I don’t see online assessment as a panacea;
• Some learning outcomes are easier than others to
assess in this way;
• Free text questions require students to construct a
response, but there still need to be definite ‘right’ and
‘wrong’ answers (though not necessarily a single right
answer);
• However, online assessment provides instantaneous
feedback and has been shown to be more accurate than
human markers. It can free up human markers for other
tasks. It also has huge potential for diagnostic use.

13. Useful links
PMatch demonstration
https://students.open.ac.uk/openmark/omdemo.pm2009/
‘Are you ready for S104?’ (diagnostic quiz, showing
a range of question types)
https://students.open.ac.uk/openmark/science.level1ayrf.s
104/
OpenMark examples site
http://www.open.ac.uk/openmarkexamples/index.shtml
COLMSCT iCMA initiative
http://www.open.ac.uk/colmsct/projects/sallyjordan

14. For further information:
• Jordan, S. & Mitchell, T. (2009) E-assessment for
learning? The potential of short free-text questions with
tailored feedback. British Journal of Educational
Technology, 40, 2, 371-385.
• Butcher, P.G. & Jordan, S.E. (2010) A comparison of
human and computer marking of short free-text student
responses. Computers & Education, 55, 489-499.

15. Sally Jordan
Staff Tutor in Science
The Open University in the East of England
Cintra House
12 Hills Road
Cambridge
CB2 1PF
s.e.jordan@open.ac.uk
website: http://www.open.ac.uk/science/people/people-
profile.php?staff_id=Sally%26%26Jordan
blog: http://www.open.ac.uk/blogs/SallyJordan/