Professor in the School of Medicine & Dean of Teaching and Learning, International Medical University, Kuala Lumpur, Malaysia

1. Tips for educators: Lessons learnt from
the development of an online
assessment system (OAS) for health
professional programmes.
Vishna Devi V Nadarajah and Er Hui Meng
International Medical University
Kuala Lumpur
Conference on Learning, Teaching and Learning 2017

2. OUTLINE
• Context:
• INTERNATIONAL MEDICAL UNIVERSITY
• Problem:
• GROWTH AND CHANGE in ASSESSMENT MANAGEMENT
• Solution:
• ONLINE ASSESSMENTS FOR HEALTH PROFESSIONAL
PROGRAMMES
• Lessons Learnt :
• Tips for Educators
Vishna Devi V Nadarajah p2

3. IMU : Outcomes Based Education
The needs of
the health care system
Outcomes & competencies
to be achieved by the
graduate
Curriculum :
Content,
Delivery,
Assessment
Demand side
education
Faculty and available
resources
Curriculum :
Content,
Delivery,
Assessment
Supply side
education
What should be taught
Vishna Devi V Nadarajah p3

4. SPICES Model
S
P
I
C
E
S
Student-centred
Problem-based
Integrated
Community-based
Electives
Systematic
Teacher-centred
Standard programme
Information-gathering
Discipline-based
Hospital-based
Apprenticeship-based/
opportunistic
Harden et al., 1984
Vishna Devi V Nadarajah p4

5. Medical Programme: Unique Model
IMU Phase 1
5 semesters
2 intakes / year
200 / intake
Partner medical
schools
Credit transfer
Graduate with PMS degree
IMU clinical school
5 semesters
Graduate as MBBS IMU
Since 1999
Vishna Devi V Nadarajah p5

6. SOM Model of Internationalisation
Transferred to other Health Professional
Programmes
• School of Pharmacy
• School of Dentistry
• School of Health Sciences
• School of Postgraduate Studies
Vishna Devi V Nadarajah p6

7. Why Quality Matters
to IMU?
• Established to meet
healthcare needs of country
• Established by educationists
• Competitive market: need to
attract quality students/ staff
• Sustain and grow international
partnerships
• Recognition of Expertise in
HPE
• Making a positive difference in
the region
Vishna Devi V Nadarajah p7

8. The Many Uses of Assessment for
Educational Management
ASSESSMENT
Grade
Quality
assurance
Improve
Student
learning
Formative
Determine
competency
Feedback
Vishna Devi V Nadarajah p8

9. Five criterias for Assessment
(Schuwirth and Van der Vleuten, 2003)
1.reliability (the degree to which the
measurement is accurate and reproducible)
2.validity (whether the assessment measures
what it claims to measure)
3. impact on future learning and practice
4. acceptability to learners and faculty
5. costs (to the individual trainee, the
institution, and society at large).
Vishna Devi V Nadarajah p9

10. All going well with
Assessments?
• Yes but we were growing
• Student numbers
• From one programme to over 15 programmes
• Diversity of faculty
• Various assessment tools
• Workload
• Lesser growth or change in
• Individual Feedback
• Exam office staffing
• Venues
Questions for reflection?
Vishna Devi V Nadarajah p10

11. To what extent are our learning
outcomes being assessed?
 Guidelines
 Implementation
 Continuous: balance of formative and
summative
 Assessment Blueprint
 Assessment Rubrics
Vishna Devi V Nadarajah p11

12. How well did we know our
assessment tools?
Assessment
tool
Competency
Assessed
Reliability Validity Impact on
Learning
Consequential
Validity (Van de
Vleuten and
Schuwirth,
2005)
MCQ
Mini -CEX
OSCE
KFA
Laboratory
/Reflective
/Community
reports, etc
Knowledge
Skills
Attitude
Vishna Devi V Nadarajah p12

13. Do we have the right resources?
 Faculty expertise and training
 Skills assessments; workplace vs stimulations
 Inter-professional assessments
 Support for examination processes
 Support for our professional staff in exam
office?
Vishna Devi V Nadarajah p13

14. Who are involved?
Vishna Devi V Nadarajah p14

15. AN URGENT NEED TO
SUPPORT AND
ENHANCE ASSESSMENTS
Vishna Devi V Nadarajah p15

16. Online Assessment System (OAS) requirements
An online assessment system that is reliable, secure and valid - cater for
a range of assessment tools (listed above) , items can include images or
video clips.
Allows storage of items in a secure question bank that is searchable by
keywords- items can be imported and exported easily to other common
word processing softwares (eg Microsoft Word, Excel).
Allows examinations to be constructed according to learning outcomes,
difficulty levels and matrix, and can have filter view
A compatible system with the university’s governance system for
assessment, which includes submission of questions, vetting processes,
examination construction, standard setting and post-item analysis
(psychometric analysis) for QA purpose.
Vishna Devi V Nadarajah p16

17. OAS requirements
Online institution of tests, where questions and options (for MCQs) can
be reshuffled
Accessible on and off-campus with security features, and allows online
grading according to assessment rubrics/checklists and multiple grading
(ie double marking), which will reduce manual processing and
transcription errors.
Generates comprehensive performance reports (including longitudinal
performance) for examination coordinators for QA purpose and to
inform the T-L strategies
Provides personal performance and immediate feedback to students. It
has an alert system to identify weak performers via
continuous/formative assessments to facilitate early remediation.
An efficient and secure data management process that can integrate
with Student Lifecycle System
Vishna Devi V Nadarajah p17

18. Scope of OAS
• User account set up
• Assigning module team
• Exam scheduling
• Item banking
• Test construction
• Vetting
• Test taking
• Results processing and reporting
• Approval
• Publishing results
Vishna Devi V Nadarajah p18

19. OAS Current Status
Started 2 years ago and initiatives still ongoing for
• Faculty training which is resource intensive
• Learning spaces renovation and hardware
acquisition
but
• Full launch and implementation for OBA
questions (70%) with existing resources
• Fine tuning for other assessment tools
Vishna Devi V Nadarajah p19

20. The lessons learnt for
Educators
Successful implementation of OAS depends on:
1. System design
2. Functionality of features
3. Faculty, Professional Staff and Vendor
Vishna Devi V Nadarajah p20

21. Tips for System Design
1. Philosophy of assessment
2. Meet users requirements
• Programme
• Students
• Exam office/ Academic Services
• IT
3. Comply to governance of assessment
• Assessment framework
• MQA
4. Security
Vishna Devi V Nadarajah p21

22. Tips for Functionality of
features
1. Ensure features are working properly
(process) to produce the end results.
2. Test processes with UAT, Mock
Assessments
3. Get relevant stakeholders in testing
processes
4. Hardware Standardisation
5. Implement a Tracking System
Vishna Devi V Nadarajah p22

23. Tips for People Management
1. Core Team on same page
2. Continuous effort for to get buy in and
ownership
3. Documentation for clarity and transfer
4. Expect Staff Turnover and PIC changes
5. Managing changes user requirements
6. Vendor as partner in development
Vishna Devi V Nadarajah p23

24. The core team :
• Assoc. Prof. Dr Er Hui Meng
• Prof Vishna Devi Nadarajah
• Prof Ammu Radhakrishnan
• Pn Noraidah Yusoff (Registrar)
• Ms Kelly Loh (Exam office)
• Ms Chin (IT)
Vishna Devi V Nadarajah p24