1.
Quality frameworks for e-learning
Jon Rosewell, The Open University (UK)
INTERNATIONAL SEMINAR ON OPEN AND DISTANCE EDUCATION (SIEAD-BR 2018)
Contributions from Open and Distance Education to Higher Education Quality: present and future
22nd October 2018

2.
174,000 students
Formal, paid-for, degrees
Supported distance learning
Open access
50 years (nearly!)
60 million visitors
Non-formal, free
OER, open courseware
Since 2006 (open2.net 1999)
8 million participants
MOOC platform
Many HE partners
Open courseware platform
CC-BY-SA-NC
Broadcast TV partner
OU/BBC audience:
265 million views
Informal learning

3.
Rise of e-learning?
• Global demand for HE
• MOOCs
• Blended learning on campus
• Society needs life-long learning –
upskilling, reskilling
• Short learning programmes?
UK undergraduate numbers
Universities UK, Patterns and trends in UK higher education 2018

4.
Quality

5.
What do we mean by ‘quality’ in HE?
• Compliance & consumer protection
– Accreditation
– Guarantee of uniform standards
• Reputation
– Recruit good students, produce good graduates
• Quality enhancement / Process improvement
– Institutional mission
– Stakeholder engagement
– Measures of added value (‘learning gain’)

6.
Approaches to QA in e-learning
• Compliance or enhancement?
• Process or product?
• Input elements?
• Pedagogical models?
• Outcome measures?
• Self-assessment or external review?
• Scorecard? Benchmarking against others?
Holistic: emphasis on process & context as well as product

7.
Quality frameworks
ICDE report on quality models 2015
• Review of quality frameworks
• Ideally:
• Multifaceted – many measures, holistic
• Dynamic – flexible when technology changes
• Mainstreamed – improvement through individuals
• Representative – balance stakeholders
• Multifunctional – e.g. external recognition, plan for
improvement, create quality culture
Ebba Ossiannilsson, Keith Williams, Anthony F. Camilleri, and Mark Brown (2015)
Quality models in online and open education around the globe: State of the art and
recommendations, ICDE Report http://www.icde.org/quality

8.
A generic framework for QA in HE
Ebba Ossiannilsson, Keith Williams, Anthony F. Camilleri, and Mark Brown (2015)
Quality models in online and open education around the globe: State of the art and
recommendations, ICDE Report http://www.icde.org/quality

9.
European Standards & Guidelines (ESG)
and e-learning
1.1 Policy for QA
1.2 Design and approval of programme
1.3 Student-centred learning, teaching & assessment
1.4 Student admission, progression, recognition & certification
1.5 Teaching staff
1.6 Learning resources and student support
1.6 Information management
1.8 Public information
1.9 Ongoing monitoring and periodic review
1.10 Cyclical external quality assurance

10.
ENQA: Considerations for QA of e-learning
• Recently published!
• Supplement to ‘European Standards and Guidelines’ 2015
• Additional guidance and indicators
Huertas et al (2018) ENQA Occasional Papers, No. 26
https://enqa.eu/index.php/publications/papers-reports/occasional-papers/

11.
E-xcellence

12.
E-xcellence label
http://e-xcellencelabel.eadtu.eu/
Manual

13.
Organisation of resources
Strategic Management a high level view of how the institution plans its e-
learning
Curriculum Design how e-learning is used across a whole programme
of study
Course Design how e-learning is used in the design of individual
courses
Course Delivery the technical and practical aspects of e-learning
delivery
Staff Support the support and training provided to staff
Student Support the support, information and guidance provided to
students

14.
Sample benchmark
Curriculum design
8. …
9. Curricula are designed to enable participation in academic
communities via social media tools. These online
communities provide opportunities for collaborative
learning, contact with external professionals and
involvement in research and professional activities.
10.…

15.
Sample indicators
• There are institutional policies relating to the provision of online community spaces for student-
student and student-teacher interactions.
• Curriculum designers specify clearly the educational role that student-student interaction plays
in their programmes.
• Criteria for the assessment of student online collaboration exist and are applied consistently
across programmes and courses.
At excellence level
• Teaching staff are supported by formal and informal staff development activity in the use of
online tools for community building.
• The institution works closely with professional bodies in the development of online professional
communities.
• Innovative assessment approaches, such as online collaborative work, peer assessment and self-
assessment, form a part of the institution’s practice in this area.

16.
Benchmarking as quality enhancement tool
• Statement of best practice
– Suggested indicators
• Collecting evidence
– Can be specific to each university
• Identification of weaknesses & strengths
• …leading to roadmap of actions for improvement

17.
Different ways to use E-xcellence
• Informal self-assessment using QuickScan
– Identify ‘hot’ and ‘cold’ spots
• Full internal self-assessment
– Stakeholders collect evidence
– Prepare roadmap of improvement actions
• Integrate with institutional process
– Embed selected benchmarks in internal process
• EADTU E-xcellence Associates Label
– Self-assessment, roadmap, external review
 recognition by EADTU
NB: Resources such as
manual and benchmarks
are freely available!

18.
Lessons learned

19.
Participant feedback
• Framework
– Quickscan is valuable to structure discussion
– Completeness of the framework is appreciated
• Team working / stakeholders
– People exchange perspectives with other departments
• External perspective
– Exchange of experience between the evaluators and staff was valuable
– New ideas surfaced for course design
• Reflection
– A valued ‘moment of reflection’ on quality
– People become aware of choices and implementations
– Gives insight into strengths and weaknesses
• Analysis
– Opportunity to formulate e-learning policy
– Provides foundations for decision making

20.
Issues when introducing e-learning

21.
Detailed issues
• Workload management (of staff)
• E-learning strategy
• Academic communities / social media
• Interactivity / e-learning tools
Non-issues:
• Reliability / performance (of VLE)
• Student support generally
– Not a ‘problem’ but much activity

22.
MOOC quality

23.
Why worry about MOOC quality?
Students – know what they are committing to
Employers – recognition of content and skills
Authors – personal reputation, 'glow' of success
Universities / providers – brand reputation
Funders – philanthropists, government, investors
Quality agencies – on behalf of all above

24.
Are MOOCs different from e-learning?
• MOOC vs Higher Education e-learning
– Short, free, no entry requirements
• MOOC participants
– Motivations differ from degree students
– Completion may not be not their goal
But a MOOC is a Course so maybe it should be judged like any
other HE course?

25.
OpenupEd Quality Label
• Derived from E-xcellence
– Lightweight process
• Self-assessment
• Formal label
– External review
www.openuped.eu/quality-label

26.
OpenupEd MOOC features
• Openness to learners
• Digital openness
• Learner-centred approach
• Independent learning
• Media-supported interaction
• Recognition options
• Quality focus
• Spectrum of diversity

27.
OpenupEd MOOC benchmarks
• Derived from E-xcellence benchmarks
• For the institution:
– To be checked every 3-5 years
– 21 benchmark statements, in six groups:
Strategic management, Curriculum design, Course design, Course delivery, Staff support,
Student support
• For the course:
– To be checked for each MOOC
– 11 benchmark statements

28.
Benchmarks – course level
22 A clear statement of learning outcomes for both knowledge and skills is provided.
23 There is reasoned coherence between learning outcomes, course content, teaching and
learning strategy (including use of media), and assessment methods.
24 Course activities aid participants to construct their own learning and to communicate it to
others.
25 The course content is relevant, accurate, and current.
26 Staff who write and deliver the course have the skills and experience to do so successfully.
27 Course components have an open licence and are correctly attributed. Reuse of material is
supported by the appropriate choice of formats and standards.
28 The course conforms to guidelines for layout, presentation and accessibility.

29.
Quick scan

30.
NA: Not achieved
PA: Partially achieved
LA: Largely achieved
FA: Fully achieved
Quick scan

31.
OL: Openness to learners
DO: Digital openness
LC: Learner-centred approach
IL: Independent learning
MI: Media-supported interaction
RO: Recognition options
QF: Quality focus
SD: Spectrum of diversity
Quick scan

32.
MOOC case study: OU + FutureLearn
A representative Open University MOOC … published on FutureLearn
• Evidence for OpenupEd features and benchmarks
• Quality emerges from joint efforts of OU (university) &
FutureLearn (platform provider)
• Holistic approach:
• Institutional and course level
• Process as well as product
• Structures and processes embed a concern for quality
throughout development, delivery and evaluation
Jansen, D., Rosewell, J., & Kear, K. (2017). ‘Quality Frameworks for MOOCs.’ In: M. Jemni, Kinshuk, & M. K. Khribi (Eds.), Open
Education: from OERs to MOOCs, 261–281. Springer http://oro.open.ac.uk/47595/

33.
Learning design

34.
Learning design – module map

35.
Learning design – activity classes
Assimilative read, watch, listen, think about, observe
Finding & handling info list, analyse, collate, find, select, manipulate
Communicative communicate, debate, discuss, collaborate, present
Productive build, write, make, design, construct, produce, draw
Experiential practice, apply, experience, investigate, perform
Interactive/adaptive explore, experiment, improve, model, simulate
Assessment write, demonstrate, critique, peer review, self-assess, receive feedback

36.
Learning design – activity planner

37.
Learning design – in practice
Toetenel, Lisette and Rienties, Bart (2016). Learning Design – creative design to visualise learning activities. Open Learning, 31(3) pp. 233–244.

38.
Learning design – in practice
Toetenel, Lisette and Rienties, Bart (2016). Learning Design – creative design to visualise learning activities. Open Learning, 31(3) pp. 233–244.

39.
What students want – and what they need
“Student satisfaction is “unrelated” to learning behaviour and
academic performance, a study has found.
[…] while students dislike collaborative learning, they are
more likely to pass if they take part in it” (Times Higher Education, Feb 12th
2018)
From an analysis of 100,000 students
on 151 modules
More at Bart Rientes, OU Inaugural Lecture

40.
How does student satisfaction relate to module performance?Satisfaction
Students who successfully completed module
Slide from Bart Rienties Inaugural lecture

41.
Summary

42.
In summary…
• A quality framework should underpin e-learning provision
– to help create a quality culture
– that is more likely to produce quality e-learning
– and quality enhancement
• There is no simple recipe, but…
– Work in a module team
– Think about learning design
– Think about student support

43.
THANK YOU
Jon.Rosewell@open.ac.uk