This document discusses quality assurance (QA) schemes for e-learning. It outlines several international QA models and labels, including UNIQUe, E-xcellence, ECBCheck, SEVAQ+, Procert, and epprobate. The document then focuses on adapting the New Zealand e-learning Maturity Model (eMM) for use in Western Balkan countries. eMM provides a framework for evaluating e-learning processes in the areas of learning, development, support, evaluation, and organization. It would require adaptation to the legislative and accreditation contexts of the Western Balkan region. The document explores customizing eMM's questionnaires and criteria for a self-evaluation of e-learning quality assurance practices

1. QA schemes
Marjan Miloshevic
prof. d-r Suzana Loshkovska
Ohrid, 14.09.2013

2. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
OUTLINE
1
• Quality of E-learning
2
• E-Learning schemes and models
3
• eMM adaptation

3. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
E-Learning
Formal
educatio
n
NON
FORMAL
EDUCATIO
N
LLL
3
E-Learning – omnipresent paradigm
http://www.badennet.org

4. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
The standard quality assurance (QA) procedures are
not sufficient
QA in e-learning is generally not a priority issue,
even for QA agencies
Numerous international projects were conducted to
form a comprehensive, yet usable framework for
quality assurance of learning.
Specialized organizations have developed their own
benchmarking procedures and tools and
established labels as brands
4
The issue of quality
http://www.badennet.org

5. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
 There are several definitions of quality of e-
learning, and each depends on its scope, objectives,
focus and the methodology of the quality approach.
 In this context, quality of e-learning is related to all:
processes, products and services for learning,
education and training supported by the use of
information and communication technologies
5
Quality of E-Learning – The Terms
http://www.badennet.org

6. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
QUALITY ASSURANCE IN HIGHER EDUCATION
A QA system documents procedures with the aim of ensuring that
the overall process meets specified objectives and to demonstrate
that quality is a managed outcome.
Quality assurance is retrospective and assures quality by
requiring conformity to externally imposed standards.
In e-learning, QA has to assure that the quality of student learning
is at least equivalent to that in face-to-face teaching/learning

7. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
7
International Quality Schemas
Label name Level Review process Type Reviewers
UNIQUe Universities Publicly documented, on site Quality mark Internal
E-xcellence University
program
Publicly documented; on-line
and on-site (optionally) Benchmark Internal
ECBCheck University
program
Documents and
accompanying web-site are in
construction phase
Quality Mark Internal
SEVAQ+
Universities,
Vocational
studies
Supporting documents and
web-site are are available; on-
line
Self-evaluation
tool
Stakeholders
(teachers,
students, ...)
Procert
Courseware
for IT
specialists
Documents are not publicly
available Benchmark N/A
epprobate General
courseware
Quality grid is public; review
conducted on-line Quality Mark Internal and
external
http://www.badennet.org

8. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
eMM

9. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
WB model – requirements
• to provide guidelines for everyone involved in quality assurance
process in HE institutions with technology-supported learning.
• to support evaluation of all forms of technology-supported learning
• distance learning, online learning, online education, distance education,
technology-enhanced education, flexible learning, flexible education, IT
supported education.
• to enable easy adaptation to different approaches of e-learning
implementation at the WB universities, different organization of QA
bodies at universities, different self-evaluation procedures, etc.

10. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Issues…
for DL programs, QA system have to ensure that the quality of learning is at
least equal to the learning in F2F programs
quality aspects for DL need to be integrated into existing QA systems, taking
into account that QA in DL and QA in F2F learning are not exactly the same
competences, resources, technology support in the DL program need to be
guaranteed
internal working methods need to be adapted to the special conditions which
apply for QA in DL

11. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Main question…
To develop a new
model/ benchmark/
framework!
To adapt an
existing one!

12. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
The choice of the model for WB
ELQ (Sweden)
RFDQ (ISO/IEC 19796-1)
E-xcellence+ (EU)
NADE (Norway)
eMM (New Zelland),
Pick&Mix (UK),
ECTI (JISC, UK),
SEVAQ+ (EU)
...
• The following models/frameworks are explored

13. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Selection of eMM
eMM is most
suitable for
the current
state in WB
countries
legislation in
WB countries
Accreditation
practice for
DL programs
in WB
QA systems in
WB

14. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Processes and their categories of eMM

15. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Learning: processes that directly impact on
pedagogical aspects of DL
L1 • Learning objectives guide the design and implementation of courses
L2
• Students are provided with mechanisms for interaction with teaching staff and other students
L3
• Students are provided with DL skill development
L4
• Students are provided with expected staff response times to student communications
L5
• Students receive feedback on their performance within courses
L6
• Students are provided with support in developing research and information literacy skills
L7
• Learning designs and activities actively engage students
L8
• Assessment is designed to progressively build student competence
L9
• Student work is subject to specified timetables and deadlines
L10
• Courses are designed to support diverse learning styles and learner capabilities

16. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Development: processes surrounding the creation
and maintenance of DL resources
D1
• Teaching staff are provided with design and development support when engaging in e-learning
D2
• Course development, design and delivery are guided by e-learning procedures and standards
D3
• An explicit plan links e-learning technology, pedagogy and content used in courses
D4
• Courses are designed to support disabled students
D5
• All elements of the physical e-learning infrastructure are reliable, robust and sufficient
D6
• All elements of the physical e-learning infrastructure are integrated using defined standards
D7
• E-learning resources are designed and managed to maximise reuse

17. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Support: processes surrounding the support
and operational management of DL
S1
• Students are provided with technical assistance when engaging in DL
S2
• Students are provided with library facilities when engaging in DL
S3
• Student enquiries, questions and complaints are collected and managed formally
S4
• Students are provided with personal and learning support services when engaging in DL
S5
• Teaching staff are provided with DL pedagogical support and professional development
S6
• Teaching staff are provided with technical support in using digital information created by
students

18. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Evaluation: Processes surrounding the evaluation and
quality control of e-learning through its entire
lifecycle
E1
• Students are able to provide regular feedback on the quality and
effectiveness of their e-learning experience
E2
• Teaching staff are able to provide regular feedback on quality and
effectiveness of their e-learning experience
E3
• Regular reviews of the e-learning aspects of courses are conducted

19. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Organisation: processes associated with institutional
planning and management
O1
• Formal criteria guide the allocation of resources for DL design, development and delivery
O2
• Institutional learning and teaching policy and strategy explicitly address DL
O3
• DL technology decisions are guided by an explicit plan
O4
• Digital information use is guided by an institutional information integrity plan
O5
• DL initiatives are guided by explicit development plans
O6
• Students are provided with information on DL technologies prior to starting courses
O7
• Students are provided with information on DL pedagogies prior to starting courses
O8
• Students are provided with administration information prior to starting courses
O9
• DL initiatives are guided by institutional strategies and operational plans

20. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Dimensions of processes
Delivery - the creation and provision of process outcomes.
Planning - the use of predefined objectives and plans in
conducting the work of the process.
Definition - the use of institutionally defined and documented
standards, guidelines, templates and policies during the process
implementation.
Management - how the institution manages the process
implementation and ensures the quality of the outcomes.
Optimisation - the extent an institution is using formal
approaches to improve the activities of the process.

21. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
The questionnaire
For each process several indicators (practices/items) are defined
For each indicator (practices/items) at least one question is defined
Each question is connected to one dimension of the process,
• P1 – delivery, P2 – planning, P3 – definition, P4 – management, P5 - optimization
For example, the practice/item the following answers are possible for each question
• "not adequate", "partially adequate", "largely adequate", "fully adequate"
• "not assessed“
Original version of eMM: over 1000 questions!

22. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
The questionnaire
The format of the questionnaire is uniform for
all eMM process categories and processes:
<Definition of the process category in eMM>
<process label> <process definition>
<practice/item label> <related assessment question for the practice/item>
<practice/item label> <related assessment question for the practice/item>
...
<process label> <process definition>
<practice/item label> <related assessment question for the practice/item>
<practice/item label> <related assessment question for the practice/item>
...
…

23. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
The questionnaire (example)
• Learning:
– L1. Learning objectives guide the design and
implementation of courses
• L1.P1. Does course documentation include a clear
statement of learning objectives? Are course workload
expectations and assessment tasks consistent with course
learning objectives?
(not assessed / not adequate / partially adequate / largely
adequate / fully adequate)
• L1.P2. ....

24. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Practice
http://www.educationcounts.govt.nz/publications/ict/58139

25. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Practice
Petch, J et al (2007) “Piloting a Process Maturity Model as an e-
Learning Benchmarking Method” The Electronic Journal
of e-Learning Volume 5 Issue 1, pp 49 - 58, available online at
www.ejel.org

26. 511126-TEMPUS-1-2010-1-RS-TEMPUS-SMGR
University
of Kragujevac
Thank you for your attention!