DOHERTY, Iain (eLearning Pedagogical Support Unit, Center for the Enhancement of Teaching and Learning. The University of Hong Kong) http://citers2012.cite.hku.hk/en/paper_521.htm

1. Evaluating the Impact of an eLearning
Strategy on the Quality of Teaching and
Learning.
Dr. Iain Doherty
Associate Professor
Director eLearning Pedagogical Support Unit
Centre for the Enhancement of Teaching and
Learning
13th May 2012

2. Overview
• HKU’s eLearning Strategy.
• Quantitative reporting.
• What is quality in learning?
• What is quality in eLearning.
• What would a program of evaluation look like?
• Concluding remarks.
2

3. eLearning Strategy Level 2
3

4. eLearning Strategy Level 3
• “Innovative pedagogy, curriculum design and
assessment are brought to new heights by
technology, including, but not restricted to,
internationalization of the curriculum, collaborative
teaching and learning within HKU courses and with
overseas universities, integration of campus-based
and experiential learning, involvement of employers
and community partners in the learning processes”.
4

5. eLearning Strategy Level 3
5

6. eLearning Strategy Level 3
6

7. Faculty Moodle Statistics
• Right now the reality is Moodle!
No. of Moodle courses
Faculty/Others Total Available Created Enabled Active
Business and
Economics 821 265 87 82
1. “Total available” - the course templates created based on SIS data or the courses created
explicitly as per requests from teachers
2. “Created” - courses that their teachers have created and can start editing its eLearning materials
3. “Enabled” - courses that their teachers have granted students’ access
4. “Active” – courses with both teacher logins > 5 and student logins > 5
7

8. Moodle Activity
• Moodle Activity Levels
– Level 1 - one-way transmission of information from
teacher to students – the data is gathered as "the total
count of Moodle files/ folders in the Moodle courses"
– Level 2 - two-way interaction between teachers &
students – total count of Moodle assignment, quizzes
and choices in the Moodle courses
– Level 3 - three way interaction between teacher &
students and among students – total count of Moodle
forum, chat and wikis in the Moodle courses
8

9. Faculty Moodle Statistics
• The news forum is created automatically which may
account for the level 1 and level 3 activities. So the
only “real figure” may be levels 1, 2 and 3.
Number of active Moodle courses having
only only only Level 1 Level 1 Level 2 Level 1,
Level 1 Level 2 Level 3 and and and Level 2
activities activities activities Level 2 Level 3 Level 3 and
Faculty
activities activities activitiesLevel 3
/Others activities
B&E 0 0 0 0 48 0 34
9

10. Quantitative Reporting
• “Quantitative Performance Indicator (PI) data has a
limited role to play in determining impact of new
learning and teaching practices on the learner
experience. PIs are ‘input’ and ‘output’ measures and
TESEP [Transforming and Enhancing the Student
Experience through Pedagogy] found that they were
best used as ‘diagnostic tools’ which could be used
on a number of levels.”
10

11. Quantitative Reporting
• These statistics tell us about activity levels in Moodle.
• The statistics do not tell us anything about the quality
of eLearning e.g.
– A lecturer on any particular course might be delivering
all the course content (level 1) via Moodle.
– The same lecturer might be engaging students with
multiple choice assessments (level 2) to test
understanding.
• What does this tell us about quality?
– Nothing!
11

12. Quality Learning
• When we say that a course is a quality course “we”
are broadly making a judgment that the course is fit
for purpose where the purpose is institutional
purpose.
• For example, university which efficiently helps
students to achieve the educational aims and
learning outcomes can be considered as a quality
institution.
• The same is true at a course level.
12

13. Quality Learning
• Quality also has something to do with the student
learning experience.
• In other words being fit for purpose is a necessary
but not sufficient condition of a quality course.
• The reason for this is that students may achieve
educational aims but have a very negative
experience on the course.
• So, we need to include some criteria referenced to
experience.
13

14. Quality in Learning
• At HKU the criteria for a quality course are given in
the Student Experience of Teaching and Learning
(SETL) questionnaire and the Student Learning
Experience Questionnaire (SLEQ).
• Both questionnaires consist of items that relate to
achieving course aims, developing generic skills /
attributes and having a positive experience on a
course.
• So a quality course is a course that is rated positively
with respect to these aspects.
14

15. Quality in eLearning
• If we know what constitutes quality in teaching and
learning we can ask about whether things are
different with eLearning.
• The short answer is “no” because at a macro level
the teaching and learning conditions are the same:
– The course must enable students to achieve ILOs;
– The course must develop generic attributes / skills; and
– Students must have a positive experience on the
course.
15

16. Quality in eLearning
• “I believe the distinction between face-to-face and
online will soon merge in both quality standard setting
and practice. As pedagogy and learning needs drive
educational design, every possible mutation of
physical and virtual meetings of minds will be created
and be grounded in pedagogical purpose. It is a
grand time to be an educator” (Frydenberg, 2002).
• There is an important point here – pedagogy and
learning drive technology use – but this is not the
whole picture for quality in eLearning.
16

17. Quality in eLearning
• Things are different at the micro level because there
is a question of the use of technologies in teaching
and learning:
– Technical infrastructure;
– Learning management system;
– Technical support for students;
– Use of additional technologies;
– Use of a variety of media;
• The big question, “Did eLearning help?”
17

18. Quality in eLearning
• The following three questions have been put forward
for use in SLEQ:
– The Learning Management System Moodle was easy
to access and use;
– eLearning has been effective in aiding learning in
courses in my discipline; and
– eLearning has been effective in aiding communication
in courses in my discipline.
• No more than 3 questions because SLEQ is already
very large.
18

19. Quality in eLearning
• What would a more robust quality assurance
program look like?
• There are issues here:
– A robust quality assurance program would be much
more comprehensive in terms of the range of
questions asked about eLearning;
– A robust quality assurance program would move
beyond questions that ask about “perceptions”;
– The program would look for evidence of e.g. improved
communication, improved understanding.
19

20. Concluding Remarks
• There has to be a program of work if we want actual
evidence that the eLearning strategy has resulted in
improved quality in teaching and learning;
• Funding would be nice because in the end showing
actual gains will require a lot of work.
• Don’t really see this program of work as optional
because without it what do we have in terms of
evidence for effectiveness of the eLearning strategy?
20

21. References
• European Association of Distance Teaching
Universities (EADTU). (2007). Quality Manual for E-
learning in Higher Education,. European Association
of Distance Teaching Universities (EADTU).
Retrieved from
http://www.eadtu.nl/e-xcellenceQS/files/members/E-xcelle
21

22. References
• Frydenberg, J. (2002). Quality Standards in
eLearning: A Matrix of Analysis. The International
Review of Research in Open and Distance Learning,
3(2), 1-15. Retrieved from
http://www.irrodl.org/index.php/irrodl/article/viewArticl
e/109
22

23. References
• Herrington, A., Herrington, J., Oliver, R., Stoney, S.,
& Willis, J. (2001). Quality Guidelines for Online
Courses: The Development of an Instrument to Audit
Online Units. In G. Kennedy & M. Keppell (Eds.),
Meeting at the Crossroads.18th Annual Conference
of the Australian Society for Computers in Learning in
Tertiary Education. (pp. 263-270). Melbourne,
Australia. Retrieved from
http://www.ascilite.org.au/conferences/melbourne01/
pdf/papers/herringtona.pdf
23

24. References
• TESEP (Transforming and Enhancing the Student
Experience through Pedagogy). (2007). TESEP: Re-
thinking Quality Enhancement (pp. 1-4). Edinburgh,
UK. Retrieved from http://
www2.napier.ac.uk/transform/rethinking_qe.htm
24