Towards quality in e learning quality assurance - epprobate international courseware label-24-09-2012
TOWARDS QUALITY IN E-LEARNING QUALITY
ASSURANCE - EPPROBATE INTERNATIONAL
Technical Faculty Čačak - University of Kragujevac, Čačak, Serbia
Faculty of Computer Science and Engineering Skopje, Macedonia
Quality of E-learning
Certification approaches and quality schemes
DL@WEB – BADEN
o Aim, objectives
o Self-assessment document
E-Learning – all present
The issue of quality
The standard quality assurance (QA) procedures
are not sufficient.
QA in e-learning is a non issue for many, especially
for the QA agencies
Numerous international projects were conducted
to form a comprehensive, yet usable framework for
quality assurance of learning.
Specialized organizations have developed their own
benchmarking procedures and tools and
established labels as brands
Quality of E-Learning – The
There are several definitions of quality of e-
learning, and each depends on its
scope, objectives, focus and the methodology
of the quality approach.
In this context, quality of e-learning is related
to all: processes, products and services for
learning, education and training supported by
the use of information and communication
Issues of internationality
an unified label
International Quality Labels
Label name Level Review process Type Reviewers
UNIQUe Universities Publicly documented, on site Quality mark Internal
Publicly documented; on-line
and on-site (optionally) Benchmark Internal
accompanying web-site are in
Quality Mark Internal
Supporting documents and
web-site are are available; on-
Documents are not publicly
available Benchmark N/A
Quality grid is public; review
conducted on-line Quality Mark Internal and
BADEN – Gathering Western
Balkan for Improved Education8
BAlkan Distance Education Network
affirmation and use of professional knowledge via e-
learning and distance education
Еducation аccessible to all, no matter of place
Improvement of academic space in WB region
Development of informatics society in the region of Balkan
through improving and promoting quality e-learning and
Modernization of education by promoting regional and
international cooperation by connecting institutions and
individuals as well as by exchange of information in the fields
of e-learning and distance education,
Tuning into The European Higher Education Area (EHEA),
Raising the level of pedagogical and technical knowledge and
skills in the fields of e-learning and distance learning,
Strengthening of educational and civil sector for successful
application of e-learning and distance education,
Goals of BADEN (2)
Fostering and cherishing research spirit for further
development of e-learning and distance education,
Strengthening of responsibility within pedagogic practice in
the field of e- learning and distance education,
More effective employment and improvement of economic
Raising social awareness about the significance of e-learning
and distance education in all aspects of formal and non-formal
Fostering and strengthening of responsibility for individual
development and improvement of status of smaller and wider
communities through activities of protection human and
minority rights, labor, racial and sex equality and right on life in
Education through professional and research seminars and
summer schools to enable providing mastering skills and
knowledge in the fields of e-learning and distance education,
Organization of professional lectures, trainings, symposiums and
conferences with topics about e-learning and distance education
Promotion of distance learning through cooperation and
participation on manifestations of domestic, regional and
international importance, competitive, propaganda and
Participation in realization of projects at national, regional and
international level as well as in other fields which have the need
for implementing this kind of education, independently or
collectively, applying at funds in the country and abroad,
Promoting and placing of innovative and creative professional
and research solutions,
Carrying out commercially readymade
solutions, individual courses or moderation services in
the fields of e-learning and distance education,
Publishing activity from the primary and related fields
which include application of e-learning techniques to
realize their activities,
Creating and continual improvement of electronic
knowledge basis from the fields of e-learning and
Promotion of the lifelong learning concept by applying e-
learning and distance education,
Creating and realizing accredited programs for
professional training of the employed in education.
Self-paced e-Learning products and services are
mainly based on courseware.
At present, the worldwide market reached $32.1 billion -
with a five-year compound annual growth rate of 9.2%
according to Ambient Insight!
There is a huge provision of eLearning courseware in
Every day different kind of new material like Web Based
Training (WBT), Computer Based Training (CBT), Game
Based Learning (GBL), Videos, Open Educational
Resources (OER), Mobile Learning Apps etc. are added.
For the learner it becomes very difficult to distinguish
what is of good quality and worth buying.
epprobate (from the Latin 'approbare', to
approve) is the first international quality label
for eLearning courseware.
The quality label is an initiative of three
Learning Agency Network (LANETO),
Agence Wallonne des Télécommunication (AWT)
e-Learning Quality Service Center (eLQSC).
1. Contacting the National Partner (BADEN for Western Balkan).
2. A short review – feasibility study.
3. The National Partner gives a recommendation and sends an offer
4. Reviewers who are able to evaluate the courseware from the points of
view of eLearning pedagogy, content accuracy, technical delivery, and
acceptability by the target group are selected.
The producer is asked to help identify representatives of the target group
Payment for review is paid at this point
5. Producer completes a Self Assessment Document answering a
range of questions about the courseware and gives access to the
courseware to the review team.
1. After the reviewers have evaluated the courseware and reached an initial set of
conclusions, the head reviewer contacts the producer and sends a report to brief
the producer for an on-line meeting with the review team.
2. During the meeting the producer receives feedback from the reviewers about the
courseware in terms of the quality criteria of the epprobate Quality Grid, and has
an opportunity to provide additional information about the courseware to address
3. After this meeting the review team revises their report and sends it together with
their recommendation as to whether the label should be awarded or not to the
international awarding committee.
1. The international awarding committee is responsible
for the final decision about the awarding of the
epprobate label and it will consider the
recommendations of the review team within the wider
context of its knowledge of global e-learning
2. The producer receives formal written feedback from
the international awarding committee, and if the
courseware has been successful in its application it
will be officially labelled by epprobate.
3. Each year an epprobate award for excellence will be
awarded to the courseware judged to be most
outstanding amongst those awarded the epprobate
label during that year.
Epprobate quality grid
Each courseware is evaluated for
A. Course design
B. Learning design
C. Media design
A. COURSE DESIGN
Provision of course information, learning
objectives and instructional guidance
The courseware provides learners with all the
information needed, including clear and appropriate
learning objectives and sufficient instructional
The learning objectives, instructional strategies and
assessment processes (where provided within the
courseware) are congruently aligned with each
other, so that the instructional strategies used can be
seen to be effectively assisting learners in achieving
the intended learning objectives.
For each topic the producer fills information in the following table
This course is intended to meet the needs of students who wish to have a
basic introduction to the nature of design thinking. It is expected that the
audience will therefore be self selecting. There is no attempt made to adapt
the materials to individual needs though the activities allow for expression of
Assessment and importance Good features Issues Other comments
For each topic, each reviewer fills information in the following table
There is no intention for any single criterion to be essential, but a
courseware supplier in their self assessment document indicates to
what extent they meet a specific criteria.
To every characteristic is assigned a mark:
A - Exceeds the requirements of the criterion
B - Meets the requirements of the criterion
C - Meets some but not all of the requirements of the criterion
D - Fails to meet the requirements of the criterion
Review - example
Good features Issues Other comments
A1 The content and
activities are well
balanced and form a mix
of examples and
individual tasks, followed
by a narrative guidance.
Assessment is modelled
to be mostly self-
conducted in form of a
practical task or
B. LEARNING DESIGN
1. Learner needs
The instructional strategies meet the learner's needs, and are matched to the learner’s
profile (skill level, age group, specific constraints, etc.)
The courseware allows personalization of the learning process and supports the
development of learner autonomy.
3. Instructional strategies
The courseware provides an appropriate range of instructional strategies that motivate
and engage learners and support active learning.
The instructional strategies support learning through an appropriate range of the forms
Acquisition – where learning is done by listening to, reading, or watching materials that are
presented in such a way as to gain the attention and interest of the learner
Inquiry - where the learner is provided with a focus for enquiry, resources, and guidance for the
Practice - where the learner is provided with interactive exercises, games, opportunities to
interact with representations of the domain, or other activities that provide feedback
Communication - where the learner is encouraged and supported in communicating with other
learners (formally and/or informally) and/or with tutors (e.g. coaching)
Construction – where the learner is provided with opportunities to learn, or to demonstrate the
outcome of their learning, through producing something - such as an essay, or a design.
C. MEDIA DESIGN
1. Media integration
The utilization of media (i.e. text, pictures, audio, and
video) and tools effectively enhances comprehension of
the course content and empowers implementation of the
The courseware interface (in terms of navigation and web
design) is user friendly, corresponds to current
practice, and allows learners to efficiently monitor their
progress through the course.
1. Interoperability and technological standards
The courseware complies with common technical
standards (e.g. SCORM, IMS) thus supporting
1. Accuracy and values of content
The content is an accurate representation of the domain
from a particular perspective or range of perspectives.
Where appropriate issues of values are addressed openly.
The content is written so as avoid culturally biased terms
2. Intellectual property rights
The courseware supplier has the rights to use the
materials in the courseware. Where material is provided as
an OER the rights of use and reuse are clearly stated with
an appropriate Creative Commons license.
3. Legal compliance
The content abides by all relevant national and
international legislation with reference to content, including
laws related to bias, slander, and promotion of specific
There is a certain progress made in quality
assurance in e-learning.
The challenges remains to be faced with, such as
internationalization, distinction in QA scale, and
usage of objective and comprehensive – yet not
too complex – methodology.
Epprobate is an initiative that tries to handle
It is an international label, which engages
numerous national partners, it treats e-learning at
the courseware level, making the QA available to
different types of producers (universities,
education agencies, companies) and it uses a
methodology that compounds all vital criteria
through a well defined procedure.