Computer assisted assessment

1. Computer Assisted
Assessment
Brian Mulligan
Institute of Technology, Sligo
Education Technology - A
Practical Approach
Sligo Park Hotel
14th
May 1999

2. What this talk is about!
What is CAA?
Why would you use it?
How would you go about it?

3. Section 1
What is CAA?
Computerised ‘objective’ tests?
Questions have pre-determined correct
answers
No-subjectivity in the correction

4. What sort of questions can
we ask?
Multiple choice / response
(MCQ)
‘Hot spot’
Text input (including numbers)
Matching pairs

5. Can Objective Tests
Measure Learning?
Validity
can we assess correct learning objectives
Reliability
to what extent will different versions of the
same test give the same result.
Accuracy
how close is the result of a test to
representing the student’s knowledge or
understanding.

6. Validity
Objective testing can examine
recall
comprehension
application
Objective testing cannot test
writing or manipulative skills
creative and synthesis skills

7. Reliability
Less luck involved
a small number of longer questions is
usually replaced by a large number of
shorter questions
Consistency in marking
due to automation
Tests may be improved
tests/questions are highly re-usable
use computerised analysis of results to
identify poor questions

8. Accuracy
An objective test can be too easy or too
difficult
Tests are highly re-usable and can easily
be adjusted

9. Software
Involved
Test Creation
Software
Questions
Tests
student data
group data
schedule data
Answer Files
Reports
Scheduling
software
Test delivery
software
Analysis and
reporting software

10. Section 2
Why would you use it?

11. Advantages of Computer
Assisted Assessment
Advantages of Objective Testing
Validity
Reliability
Accuracy
Efficiency
Advantages of Computerisation
Further efficiency
may result in higher accuracy
and better reliability
Verifiability
Less stress on students

12. Uses of CAA
Summative Assessment
Formative Assessment
Continuous Assessment
Self Testing
Encouragement to work
Course customisation / control
Monitoring of Independent Student Learning

13. Section 3
How would you go about implementing it

14. Summative Testing
Aim: measure a student’s knowledge and
understanding of a course
Accuracy and reliability important.
May not be able to test all learning objectives
Test creation, delivery and unsophisticated
reporting software will suffice
You may manually make tests available for
students
on a floppy disk or on a network
Supervise students but not necessarily
simultaneously

15. Formative Testing
Assessing students during a course
Accuracy and reliability may NOT be
important
Large numbers of short tests
A good management system is generally
required
see next few slides

16. Continuous Assessment
Accurate measurement of progress
Reliable and accurate tests required
Secure supervised tests
Not necessarily simultaneous
Good scheduling software required
Automatic analysis/ reporting software
required
Make tests available on a network with
restricted access privileges

17. Self Testing
Tests that students can take at any time
Accuracy and reliability are important
Security is not an issue!
Could be made available on a network or
distributed on floppy disk
Students may retake the tests many times
A large bank of questions required
Will they be able to predict your ‘secure’
tests from these?
You may not need to store results

18. Encouragement to work
Regular tests that count towards a student’s
final mark
Accuracy and reliability NOT so important!
Secure supervised tests
Make tests available on a network with
restricted access privileges
IT Sligo have carried out a pilot study on this:
Impact of Frequent Computerised
Assessment on Student Work Rates

19. Pilot Study Conclusions
Work rates do increase!
A good test management system is required
(includes scheduling, reporting and analysis)
Students can take tests in close proximity.
All students need not be tested simultaneously.
Students like this form of assessment.
Lecturers think it is well worthwhile.
4 hours initial training.
20 minutes for weekly test preparation
15 minutes to test student on all subjects

20. Course customisation /
control
CAA may be used simply to stream students
Learning material built into tests (feedback)
Direction to further learning built into tests
May restrict students from moving on
Accuracy and reliability important
Security may also be an issue
Tests may be integrated with CAL
CAL courseware may have embedded tests
A course management system (CMS) may be
a good solution

21. Monitoring of Independent
Student Learning
Some students may cope poorly with
the responsibility
May be used to identify students falling
behind
Accuracy and reliability important
Security is also important.
Similar issues to that involved in
‘continuous assessment’
A CMS may be the best tool

22. Summary
CAA is relatively simple because of the limited
scope - especially in Summative Assessment
However, it’s scope is wider than you might
think
Many reasons to use it
In Formative Assessment, the Management of
CAA and integration into course delivery may be
bigger issues than the assessment itself