Presentation by Jon Rosewell (OUUK) during the EADTU-ENQA Peer Learning Activity in Brussels on 21-22 September 2017

1. The E-xcellence QA methodology:
Lessons learned over ten years of development and
implementation
Jon Rosewell (Open University, UK), Karen Kear(OU), Keith Williams (OU),
Covadonga Rodrigo (UNED), Angeles Sanchez Elvira Paniagua (UNED)
QA of Blended and Online Programmes
EADTU-ENQA Peer Learning Activity
Brussels, 21-22 Sept 2017

2. What do we mean by ‘quality’ in HE?
• Reputation
– Recruit good students, produce good graduates
• Compliance & consumer protection
– Accreditation
– Guarantee of uniform standards
• Process improvement
– To meet institutional mission (so internal focus)
– Stakeholder engagement
– Measures of added value (for external use?)

3. Approaches to QA in e-learning
• Compliance or enhancement?
• Norms?
• Process or product?
• Pedagogical models?
• Input elements?
• Outcome measures?
• Self-assessment or external review?
Holistic: emphasis on process & context as well as product

4. The future will be blended
campus — distance
full time — part time
face-to-face — online
synchronous — asynchronous
paper — digital
physical — virtual
Challenge: finding a ‘principled’ mix to meet needs
‘Principled’: justified on basis of research / evaluation / experience / …

5. Enablers of blended provision
Underpinned by ubiquitous access to internet through personal
devices
• Digital media / content
• VLE / LMS / registration / support
• Social media use
• Online library provision
• Open educational resources
• (MOOCs?)

6. Challenges for QA of e/b-learning
• Tension between:
– High level standards that are independent of mode
– Detailed indicators for e-learning, campus, blend…
• Tension between:
– External, national/international
– Internal
• Tension between:
– Compliance, checking, retrospective
– Enhancement, prospective
• Tension between:
– Content: components and product
– Process and context

7. E-xcellence label
http://e-xcellencelabel.eadtu.eu/
Manual
1. Strategic management
2. Curriculum design
3. Course design
4. Course delivery
5. Staff support
6. Student support

8. Organisation of resources
Strategic Management 5 benchmark statements
Curriculum Design 4 benchmark statements
Course Design 9 benchmark statements
Course Delivery 6 benchmark statements
Staff Support 6 benchmark statements
Student Support 5 benchmark statements

9. Sample benchmark
Curriculum design
8. …
9. Curricula are designed to enable participation in academic
communities via social media tools. These online
communities provide opportunities for collaborative
learning, contact with external professionals and
involvement in research and professional activities.
10. …

10. Sample indicators
• There are institutional policies relating to the provision of online community
spaces for student-student and student-teacher interactions.
• Curriculum designers specify clearly the educational role that student-student
interaction plays in their programmes.
• Criteria for the assessment of student online collaboration exist and are
applied consistently across programmes and courses.
At excellence level
• Teaching staff are supported by formal and informal staff development activity
in the use of online tools for community building.
• The institution works closely with professional bodies in the development of
online professional communities.
• Innovative assessment approaches, such as online collaborative work, peer
assessment and self-assessment, form a part of the institution’s practice in
this area.

11. E-xcellence benchmarks
• Benchmark = statement of best practice
• Benchmarking as quality enhancement tool
– Identification of weaknesses & strengths by collecting evidence
– Action plan for improvement
• Generic in form
– Resilient to developments in technology and pedagogy
• Reflect current experience and practice
– Informed by identifiable trends, evolve over time
• Complementary with OpenupEd MOOC benchmarks
• Published under Creative Commons licensing

12. Modes of use
• Informal self-assessment using QuickScan
– Identify ‘hot’ and ‘cold’ spots
• Full internal self-assessment
– Stakeholders collect evidence and prepare road map of actions
• Integrate with institutional process
– Embed selected benchmarks in internal process
• EADTU E-xcellence Associates Label
– Self-assessment and external review  recognition by EADTU
• Integrate into QA agencies formal process
– External agencies embed benchmarks in formal process
NB: Resources
such as manual
and benchmarks
are freely
available!
Seminar / workshop

13. Review of reviews
• Nineteen reviews 2009-2016
• For each benchmark, counted no. of reviews with:
– ‘Problematic’ in self-assessments
– Actions planned in road maps
• Grouped by major area (chapter of manual)

18. Emerging issues
• Workload management (of staff)
• E-learning strategy
• Academic communities / social media
• Interactivity / e-learning tools
Non-issues:
• Student support generally
• Reliability / performance (of VLE)

19. Local seminar feedback
• Framework
– Quickscan is valuable to structure discussion
– Completeness of the framework is appreciated
• Team working
– People exchange perspectives with other departments
• External perspective
– Exchange of experience between the evaluators and staff was valuable
– New ideas surfaced for course design
• Reflection
– A valued ‘moment of reflection’ on quality
– People become aware of choices and implementations
– Gives insight into strengths and weaknesses
• Analysis
– Opportunity to formulate e-learning policy
– Provides foundations for decision making
E-xcellence NEXT

20. What aspects did you like the most about the
E-xcellence seminar?
• Open discussion, sharing views and experiences
• The process of completing the Quick Scan
• Opportunity to exchange experiences between local team
and experts from other countries
• Sharing experience, revealing strengths and weaknesses
• Improvement plan
• E-xcellence tool is extremely useful for improving quality
• Commitment of participants: managers, teachers, students,
incl. representative of National accreditation agency
• Discussions that are relevant for future development
• It was an opportunity to revise our master degree program
SEQUENT

21. Has the use of the instrument had any impact
in your organisation so far?
• Discussion on some important issues was initiated, involving stakeholders
not involved before
• We have identified issues for further e-learning development
• The participation in the project has already led to adjustments of the QA
system of the university
• It has opened a dialogue between different stakeholders
• We are working on improvement on e-learning performance along the
road map
• Definitely, QA in e-learning became more important
• Since the project teachers see how to improve their courses
• Too early for an official impact, but it enhances fruitful discussions
• E-xcellence framework is very useful in the process of designing an
internal QA system on e-learning
• Not yet, some changes require decision making at the central level

22. What role could QA agencies play in relation to
QA in e-learning?
• They should regard e-learning as an indispensable element of
modern education
• Create an independent assessment of e-learning/on-line
programmes
• They should prepare recommendations in QA and e-learning
• Provide a more detailed implementation plan
• They should both play a constructive and an accreditive role in
the development of a methodology to assess quality in e-
learning
• Organise workshops on QA benchmarks
SEQUENT

23. ICDE Global Overview of Quality Models 2015
• Norm based approaches are subject to rapid obsolescence
• Staff development in institutions and agencies is a challenge
• Focus on principles and flexibility
• As systems converge, face-to-face sector can learn from
distance education sector
Ebba Ossiannilsson, Keith Williams, Anthony F. Camilleri, and Mark Brown (2015)
Quality models in online and open education around the globe: State of the art and
recommendations, ICDE Report http://www.icde.org/quality

24. Questions?
Thanks for your attention
Jon.Rosewell@open.ac.uk
George.Ubachs@eadtu.eu