This is a presentation that I gave at the CITE 2012 Research Symposium at The University of Hong Kong. Essentially the presentation makes a case for the need to look at learning experiences and learning outcomes in order to truly determine whether or not an eLearning strategy has been effective. In other words, Moodle generated data will not do the trick.

1. Evaluating the Impact of an eLearning
Strategy on the Quality of Teaching and
Learning.
Dr. Iain Doherty
Associate Professor
Director eLearning Pedagogical Support Unit
Centre for the Enhancement of Teaching and
Learning
16th May 2012

2. Overview
• HKU’s eLearning Strategy.
• Quantitative reporting.
• What is quality in learning?
• What is quality in eLearning.
• What would a program of evaluation look like?
• Concluding remarks.
2

3. eLearning Strategy Level One
• Teaching and learning is assisted by technology for
efficiency and cost-effectiveness. Students are
provided with on-line access to:
– Information regarding their programmes of study,
including administrative procedures relating to course
enrolment, assessments, degree audit, student
evaluation of teaching and learning (SETL); and
– Resources for learning, including all course outlines
and digital materials, course learning outcomes and
assessments.

4. eLearning Strategy Level One

5. eLearning Strategy Level Two
• Teaching and learning is enriched by technology
through enhancing opportunities for active learning
within and beyond the classroom, provision of links to
digital library resources, provision of just-in-time
formative and summative feedback and assessment
for learning, fostering teaching-research nexus,
establishing a closer link with schools, community
partners and employers.

6. eLearning Strategy Level 2
6

7. eLearning Strategy Level 3
• “Innovative pedagogy, curriculum design and
assessment are brought to new heights by
technology, including, but not restricted to,
internationalization of the curriculum, collaborative
teaching and learning within HKU courses and with
overseas universities, integration of campus-based
and experiential learning, involvement of employers
and community partners in the learning processes”.
7

8. eLearning Strategy Level 3
8

9. eLearning Strategy Level 3
9

10. Faculty Moodle Statistics
• Right now the reality is Moodle!
No. of Moodle courses
Faculty/Others Total Available Created Enabled Active
821 265 87 82
1. “Total available” - the course templates created based on SIS data or the courses created
explicitly as per requests from teachers
2. “Created” - courses that their teachers have created and can start editing its eLearning materials
3. “Enabled” - courses that their teachers have granted students’ access
4. “Active” – courses with both teacher logins > 5 and student logins > 5
10

11. Moodle Activity
• Moodle Activity Levels
– Level 1 - one-way transmission of information from
teacher to students – the data is gathered as "the total
count of Moodle files/ folders in the Moodle courses"
– Level 2 - two-way interaction between teachers &
students – total count of Moodle assignment, quizzes
and choices in the Moodle courses
– Level 3 - three way interaction between teacher &
students and among students – total count of Moodle
forum, chat and wikis in the Moodle courses
11

12. Faculty Moodle Statistics
• The news forum is created automatically which may
account for the level 1 and level 3 activities. So the
only “real figure” may be levels 1, 2 and 3.
Number of active Moodle courses having
only Level 1 only Level 2 only Level 3 Level 1 and Level 1 and Level 2 and Level 1, Level
activities activities activities Level 2 Level 3 Level 3 2 and Level 3
Faculty activities activities activities activities
/Others
0 0 0 0 48 0 34
12

13. Quantitative Reporting
• “Quantitative Performance Indicator (PI) data has a
limited role to play in determining impact of new
learning and teaching practices on the learner
experience. PIs are ‘input’ and ‘output’ measures and
TESEP [Transforming and Enhancing the Student
Experience through Pedagogy] found that they were
best used as ‘diagnostic tools’ which could be used
on a number of levels.”
13

14. Quantitative Reporting
• These statistics tell us about activity levels in Moodle.
• The statistics do not tell us anything about the quality
of eLearning e.g.
– A lecturer on any particular course might be delivering
all the course content (level 1) via Moodle.
– The same lecturer might be engaging students with
multiple choice assessments (level 2) to test
understanding.
• What does this tell us about quality?
– Nothing!
14

15. Quality Learning
• When we say that a course is a quality course “we”
are broadly making a judgment that the course is fit
for purpose where the purpose is institutional
purpose.
• For example, university which efficiently helps
students to achieve the educational aims and
learning outcomes can be considered as a quality
institution.
• The same is true at a course level.
15

16. Quality Learning QAC
• “The main objective of QAC quality audits is to
assure the quality of student learning in UGC-funded
institutions. The audits are intended to assure the
UGC and the public that institutions deliver on the
promises they make in their role and mission
statements. A QAC audit is therefore an audit of an
institution’s Fitness for Purpose in teaching and
learning” (Quality Assurance Council, 2008, p.10).
16

17. Quality Learning QAC
• QAC is concerned with learning outcomes:
• “the audit process is particularly concerned with the
ways institutions articulate and measure the student
learning outcomes they expect or aspire to” (Quality
Assurance Council, 2008, p.11)
17

18. Quality Learning
• Quality also has something to do with the student
learning experience.
• In other words being fit for purpose is a necessary
but not sufficient condition of a quality course.
• The reason for this is that students may achieve
educational aims but have a very negative
experience on the course.
• So, we need to include some criteria referenced to
experience.
18

19. Quality Learning QAC
• The QAC, through its audit process, aims to give
confidence, in general, to students and their parents,
employers and sponsors, that our institutions provide
a quality and internationally recognised student
learning experience (Quality Assurance Council,
2008, p.3.).
19

20. Quality in Learning
• At HKU the criteria for a quality course are given in
the Student Experience of Teaching and Learning
(SETL) questionnaire and the Student Learning
Experience Questionnaire (SLEQ).
• Both questionnaires consist of items that relate to
achieving course aims, developing generic skills /
attributes and having a positive experience on a
course.
• So a quality course is a course that is rated positively
with respect to these aspects.
20

21. Quality in eLearning
• If we know what constitutes quality in teaching and
learning we can ask about whether things are
different with eLearning.
• The short answer is “no” because at a macro level
the teaching and learning conditions are the same:
– The course must enable students to achieve ILOs;
– The course must develop generic attributes / skills; and
– Students must have a positive experience on the
course.
21

22. Quality in eLearning
• “I believe the distinction between face-to-face and
online will soon merge in both quality standard setting
and practice. As pedagogy and learning needs drive
educational design, every possible mutation of
physical and virtual meetings of minds will be created
and be grounded in pedagogical purpose. It is a
grand time to be an educator” (Frydenberg, 2002).
• There is an important point here – pedagogy and
learning drive technology use – but this is not the
whole picture for quality in eLearning.
22

23. Quality in eLearning
• Things are different at the micro level because there
is a question of the use of technologies in teaching
and learning:
– Technical infrastructure;
– Learning management system;
– Technical support for students;
– Use of additional technologies;
– Use of a variety of media;
• The big question, “Did eLearning help?”
23

24. Quality in eLearning
• The following three questions have been put forward
for use in HKUs Student Learning Experience
Questionnaire:
– The Learning Management System Moodle was easy
to access and use;
– eLearning has been effective in aiding learning in
courses in my discipline; and
– eLearning has been effective in aiding communication
in courses in my discipline.
24

25. Quality in eLearning
• What would a more robust quality assurance
program look like?
• There are issues here:
– A robust quality assurance program would be much
more comprehensive in terms of the range of
questions asked about eLearning;
– A robust quality assurance program would move
beyond questions that ask about “perceptions”;
– The program would look for evidence of e.g. improved
communication, improved understanding.
25

26. Concluding Remarks
• There has to be a program of work if we want actual
evidence that the eLearning strategy has resulted in
improved quality in teaching and learning;
• Funding would be nice because in the end showing
actual gains will require a lot of work.
• Don’t really see this program of work as optional
because without it what do we have in terms of
evidence for effectiveness of the eLearning strategy?
26

27. References
• European Association of Distance Teaching
Universities (EADTU). (2007). Quality Manual for E-
learning in Higher Education,. European Association
of Distance Teaching Universities (EADTU).
Retrieved from
http://www.eadtu.nl/e-xcellenceQS/files/members/E-xcelle
27

28. References
• Frydenberg, J. (2002). Quality Standards in
eLearning: A Matrix of Analysis. The International
Review of Research in Open and Distance Learning,
3(2), 1-15. Retrieved from
http://www.irrodl.org/index.php/irrodl/article/viewArticl
e/109
28

29. References
• Herrington, A., Herrington, J., Oliver, R., Stoney, S.,
& Willis, J. (2001). Quality Guidelines for Online
Courses: The Development of an Instrument to Audit
Online Units. In G. Kennedy & M. Keppell (Eds.),
Meeting at the Crossroads.18th Annual Conference
of the Australian Society for Computers in Learning in
Tertiary Education. (pp. 263-270). Melbourne,
Australia. Retrieved from
http://www.ascilite.org.au/conferences/melbourne01/
pdf/papers/herringtona.pdf
29

30. References
• TESEP (Transforming and Enhancing the Student
Experience through Pedagogy). (2007). TESEP: Re-
thinking Quality Enhancement (pp. 1-4). Edinburgh,
UK. Retrieved from
http://www2.napier.ac.uk/transform/rethinking_qe.htm
30