Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Quality assurance of MOOCs:
The OpenupEd quality label
Jon Rosewell, The Open University (UK)
OpenupEd week
16 April 2019
Quality in e-learning
What do we mean by ‘quality’ in HE?
• Compliance & consumer protection
– Accreditation
– Guarantee of uniform standards
• ...
Approaches to QA in e-learning
• Compliance or enhancement?
• Process or product?
• Input elements?
• Pedagogical models?
...
A generic framework for QA in HE
Ebba Ossiannilsson, Keith Williams, Anthony F. Camilleri, and Mark Brown (2015)
Quality m...
European Standards & Guidelines (ESG)
and e-learning
1.1 Policy for QA
1.2 Design and approval of programme
1.3 Student-ce...
ENQA: Considerations for QA of e-learning
• Published 2018
• Supplement to ‘European Standards and Guidelines’ 2015
• Addi...
Poll – do you use a QA process/framework?
• No
• Yes, internally defined
• Yes, defined by QA / Govt agency
Poll – why would you do more QA?
❑I want to improve my teaching
❑My boss tells me to
❑It is my job!
❑The QA agency / Minis...
E-xcellence
E-xcellence label
http://e-xcellencelabel.eadtu.eu/
Manual
Organisation of resources
Strategic Management a high level view of how the institution plans its e-learning
Curriculum De...
Sample benchmark
Course design
10. …
11. Learning outcomes determine the use of methods and
course contents. In a blended-...
Sample indicators
Indicators
• Fitness for purpose drives decisions on the selection of teaching and
learning activities. ...
Benchmarking as quality enhancement tool
• Statement of best practice
– Suggested indicators
• Collecting evidence
– Can b...
Poll – who should collect evidence?
• Course author
• Administrator
• Students
• External reviewer
• Team of stakeholders
Different ways to use E-xcellence
• Informal self-assessment using QuickScan
– Identify ‘hot’ and ‘cold’ spots
• Full inte...
MOOC quality
Why worry about MOOC quality?
Students – know what they are committing to
Employers – recognition of content and skills
Au...
Are MOOCs different from e-learning?
• MOOC vs Higher Education e-learning
– Short, free, no entry requirements
– Not accr...
OpenupEd Quality Label
• Derived from E-xcellence
– Lightweight process
• Self-assessment
• Formal label
– External review...
OpenupEd MOOC features
• Openness to learners
• Digital openness
• Learner-centred approach
• Independent learning
• Media...
OpenupEd MOOC benchmarks
• Derived from E-xcellence benchmarks
• For the institution:
– To be checked every 3-5 years
– 21...
Benchmarks – course level
22. A clear statement of learning outcomes for both knowledge and skills is provided.
23. There ...
Benchmarks – course level
29. The course contains sufficient interactivity (student-to-content or student-to-
student) to ...
Additional notes – example
31. Assessment is explicit, fair, valid and reliable. Measures appropriate to the
level of cert...
Templates – evidence
Quick scan
NA: Not achieved
PA: Partially achieved
LA: Largely achieved
FA: Fully achieved
Quick scan
OL: Openness to learners
DO: Digital openness
LC: Learner-centred approach
IL: Independent learning
MI: Media-supported in...
Whiteboard – your MOOC experience
Not achieved Fully achieved
22. Clear learning outcomes
23. Aligned LOs, content, assess...
Whiteboard – MOOC features
• Openness to learners
• Digital openness
• Learner-centred approach
• Independent learning
• M...
New checklists (OpenupEd, SCORE2020)
Checklist 1: Is it a MOOC or not?
– 14 items
Checklist 2: Quality of the design of MO...
Checklist 2: Quality of design
Dimension Criteria Level
Target group MOOCs are accessible to all people and as such variou...
Checklist 2: Quality of design
Dimension Criteria Level
Learning
activities
Activities aid participants to construct their...
Checklist 2: Quality of design
Dimension Criteria Level
Feedback
mechanism
Feedback by an academic tutor is limited and sc...
QRF from MOOC-Quality.eu
Phases / perspectives / roles
QRF phases, processes, criteria
• Analysis 7 processes 25 criteria
• Design 11 processes 54 criteria
• Implementation 6 pr...
Poll – your preference
• Few generic benchmarks supplemented by evidence
• Many detailed criteria
Learning design
What is learning design?
• A way of documenting the design of a course
• A way of thinking about & discussing design
– bef...
The 7Cs of Learning Design (Gráinne Conole)
Many activities to help teams
consider each of these stages
– Course tweet (el...
UCL Learning Designer (Diana Laurillard)
• Acquisition
• Inquiry
• Practice
• Production
• Discussion
• Collaboration
http...
UCL Learning Designer (Diana Laurillard)
https://www.ucl.ac.uk/learning-designer/
8 Learning Events (Leclercq & Poumay)
• Imitation / modelling
• Reception / transmission
• Exercising / feedback
• Explora...
First Principles of Instruction (Merrill)
– with a few more! (Margaryan)
1. Problem-centred
2. Activation
3. Demonstration...
OU Learning design – module map
OU Learning design – activity classes
Assimilative read, watch, listen, think about, observe
Finding & handling info list,...
OU Learning design – activity planner
OU Learning design – in practice
Toetenel, Lisette and Rienties, Bart (2016). Learning Design – creative design to visuali...
OU Learning design – in practice
Toetenel, Lisette and Rienties, Bart (2016). Learning Design – creative design to visuali...
In summary…
• A quality framework should underpin e-learning provision
– to help create a quality culture
– that is more l...
THANK YOU
Jon.Rosewell@open.ac.uk
Extra slides!
What students want – and what they need
“Student satisfaction is “unrelated” to learning behaviour and
academic performanc...
How does student satisfaction relate to module performance?Satisfaction
Students who successfully completed module
Slide f...
MOOC case study: OU + FutureLearn
A representative Open University MOOC … published on FutureLearn
• Evidence for OpenupEd...
Summary
In summary…
• A quality framework should underpin e-learning provision
– to help create a quality culture
– that is more l...
Checklist 1: Is it a MOOC or not?
Dimension Criteria Level
Massive Pedagogical model means effort doesn’t increase
signifi...
Quality assurance of MOOCs: The OpenupEd quality label
Quality assurance of MOOCs: The OpenupEd quality label
Quality assurance of MOOCs: The OpenupEd quality label
Upcoming SlideShare
Loading in …5
×

Quality assurance of MOOCs: The OpenupEd quality label

66 views

Published on

[Qa of moo cs openup-ed-week-2019] - Jon Rosewell

Published in: Education
  • Be the first to comment

  • Be the first to like this

Quality assurance of MOOCs: The OpenupEd quality label

  1. 1. Quality assurance of MOOCs: The OpenupEd quality label Jon Rosewell, The Open University (UK) OpenupEd week 16 April 2019
  2. 2. Quality in e-learning
  3. 3. What do we mean by ‘quality’ in HE? • Compliance & consumer protection – Accreditation – Guarantee of uniform standards • Reputation – Recruit good students, produce good graduates • Quality enhancement / Process improvement – Institutional mission – Stakeholder engagement – Measures of added value (‘learning gain’)
  4. 4. Approaches to QA in e-learning • Compliance or enhancement? • Process or product? • Input elements? • Pedagogical models? • Outcome measures? • Self-assessment or external review? • Scorecard? Benchmarking against others? Holistic: emphasis on process & context as well as product
  5. 5. A generic framework for QA in HE Ebba Ossiannilsson, Keith Williams, Anthony F. Camilleri, and Mark Brown (2015) Quality models in online and open education around the globe: State of the art and recommendations, ICDE Report http://www.icde.org/quality
  6. 6. European Standards & Guidelines (ESG) and e-learning 1.1 Policy for QA 1.2 Design and approval of programme 1.3 Student-centred learning, teaching & assessment 1.4 Student admission, progression, recognition & certification 1.5 Teaching staff 1.6 Learning resources and student support 1.6 Information management 1.8 Public information 1.9 Ongoing monitoring and periodic review 1.10 Cyclical external quality assurance
  7. 7. ENQA: Considerations for QA of e-learning • Published 2018 • Supplement to ‘European Standards and Guidelines’ 2015 • Additional guidance and indicators Huertas et al (2018) ENQA Occasional Papers, No. 26 https://enqa.eu/index.php/publications/papers-reports/occasional-papers/
  8. 8. Poll – do you use a QA process/framework? • No • Yes, internally defined • Yes, defined by QA / Govt agency
  9. 9. Poll – why would you do more QA? ❑I want to improve my teaching ❑My boss tells me to ❑It is my job! ❑The QA agency / Ministry make me ❑I want a promotion ❑Other reasons Check as many as apply
  10. 10. E-xcellence
  11. 11. E-xcellence label http://e-xcellencelabel.eadtu.eu/ Manual
  12. 12. Organisation of resources Strategic Management a high level view of how the institution plans its e-learning Curriculum Design how e-learning is used across a whole programme of study Course Design how e-learning is used in the design of individual courses Course Delivery the technical and practical aspects of e-learning delivery Staff Support the support and training provided to staff Student Support the support, information and guidance provided to students
  13. 13. Sample benchmark Course design 10. … 11. Learning outcomes determine the use of methods and course contents. In a blended-learning context there is an explicit rationale for the use of each element in the blend. 12. …
  14. 14. Sample indicators Indicators • Fitness for purpose drives decisions on the selection of teaching and learning activities. The blending is such that different methods and media are well chosen within and between courses, both in distribution over time and extent of use. At excellence level • There is extensive institutional experience of delivery using blended learning and this experience is widely shared through the organisation. • Well informed decisions on the use of teaching and learning activities are made routinely and reflect institutional policies regarding the development of learner knowledge and skills.
  15. 15. Benchmarking as quality enhancement tool • Statement of best practice – Suggested indicators • Collecting evidence – Can be specific to each university • Identification of weaknesses & strengths • …leading to roadmap of actions for improvement
  16. 16. Poll – who should collect evidence? • Course author • Administrator • Students • External reviewer • Team of stakeholders
  17. 17. Different ways to use E-xcellence • Informal self-assessment using QuickScan – Identify ‘hot’ and ‘cold’ spots • Full internal self-assessment – Stakeholders collect evidence – Prepare roadmap of improvement actions • Integrate with institutional process – Embed selected benchmarks in internal process • EADTU E-xcellence Associates Label – Self-assessment, roadmap, external review → recognition by EADTU NB: Resources such as manual and benchmarks are freely available!
  18. 18. MOOC quality
  19. 19. Why worry about MOOC quality? Students – know what they are committing to Employers – recognition of content and skills Authors – personal reputation, 'glow' of success Universities / providers – brand reputation Funders – philanthropists, government, investors Quality agencies – on behalf of all above
  20. 20. Are MOOCs different from e-learning? • MOOC vs Higher Education e-learning – Short, free, no entry requirements – Not accredited – Reputational risk • MOOC participants – Motivations differ from degree students – Completion may not be not their goal But a MOOC is a Course so maybe it should be judged like any other HE course?
  21. 21. OpenupEd Quality Label • Derived from E-xcellence – Lightweight process • Self-assessment • Formal label – External review www.openuped.eu/quality-label
  22. 22. OpenupEd MOOC features • Openness to learners • Digital openness • Learner-centred approach • Independent learning • Media-supported interaction • Recognition options • Quality focus • Spectrum of diversity
  23. 23. OpenupEd MOOC benchmarks • Derived from E-xcellence benchmarks • For the institution: – To be checked every 3-5 years – 21 benchmark statements, in six groups: Strategic management, Curriculum design, Course design, Course delivery, Staff support, Student support • For the course: – To be checked for each MOOC – 11 benchmark statements
  24. 24. Benchmarks – course level 22. A clear statement of learning outcomes for both knowledge and skills is provided. 23. There is reasoned coherence between learning outcomes, course content, teaching and learning strategy (including use of media), and assessment methods. 24. Course activities aid participants to construct their own learning and to communicate it to others. 25. The course content is relevant, accurate, and current. 26. Staff who write and deliver the course have the skills and experience to do so successfully. 27. Course components have an open licence and are correctly attributed. Reuse of material is supported by the appropriate choice of formats and standards. 28. The course conforms to guidelines for layout, presentation and accessibility.
  25. 25. Benchmarks – course level 29. The course contains sufficient interactivity (student-to-content or student-to- student) to encourage active engagement. The course provides learners with regular feedback through self-assessment activities, tests or peer feedback. 30. Learning outcomes are assessed using a balance of formative and summative assessment appropriate to the level of certification. 31. Assessment is explicit, fair, valid and reliable. Measures appropriate to the level of certification are in place to counter impersonation and plagiarism. 32. Course materials are reviewed, updated and improved using feedback from stakeholders.
  26. 26. Additional notes – example 31. Assessment is explicit, fair, valid and reliable. Measures appropriate to the level of certification are in place to counter impersonation and plagiarism. See comments to OpenupEd benchmark 29 above. The advent of digital badges (for example Mozilla open badges) provides a method of rewarding achievement that may be appropriate for MOOCs. The award of digital badges can be linked to automated or peer assessment. Digital badges have an infrastructure that verifies the identity of the holder and provides a link back to the issuer and the criteria and evidence for which it was awarded. Badges thus may provide a validated award that can be kept distinct from the HEI’s normal qualifications. See also: E-xcellence benchmark #17 Chapter 3 Course design § 2.3.1 Transferable skills § 2.4 Assessment procedures § 3.4 Assessment § 4.2.5 Online assessment
  27. 27. Templates – evidence
  28. 28. Quick scan
  29. 29. NA: Not achieved PA: Partially achieved LA: Largely achieved FA: Fully achieved Quick scan
  30. 30. OL: Openness to learners DO: Digital openness LC: Learner-centred approach IL: Independent learning MI: Media-supported interaction RO: Recognition options QF: Quality focus SD: Spectrum of diversity Quick scan
  31. 31. Whiteboard – your MOOC experience Not achieved Fully achieved 22. Clear learning outcomes 23. Aligned LOs, content, assessment 24. Activities construct learning 25. Relevant, accurate, current 29. Interactivity, active learning, self-ass. 30. Formative & summative assessment
  32. 32. Whiteboard – MOOC features • Openness to learners • Digital openness • Learner-centred approach • Independent learning • Media-supported interaction • Recognition options • Quality focus • Spectrum of diversity
  33. 33. New checklists (OpenupEd, SCORE2020) Checklist 1: Is it a MOOC or not? – 14 items Checklist 2: Quality of the design of MOOC – 26 items Checklist 3: Accessibility – 6 items Checklist 4: Technical platform and support for staff and participants – 12 items
  34. 34. Checklist 2: Quality of design Dimension Criteria Level Target group MOOCs are accessible to all people and as such various target groups are identified For each target group the needs, challenges and prior knowledge are described The description of each target group is supported by references different studies Overall goal The overall objective of the course is described in a few sentences Learning ob. The course describes a limited number of learning objectives There is a reasoned coherence between learning outcomes, course content, teaching and learning strategy (including use of Levels: Not achieved, Partially achieved, Largely achieved, Fully achieved
  35. 35. Checklist 2: Quality of design Dimension Criteria Level Learning activities Activities aid participants to construct their own learning and to communicate it to others The ‘pathways’ (activities, tasks and routes) are designed in such a way that they can be performed at different levels of difficulty or complexity, to account for the broad spectrum of participants’ knowledge and skills that are expected Various activities are proposed with different formats. For example: quizzes, peer evaluation, video conferences, activities in forums or external social networks). The course contains sufficient interactivity (learner-to-content, learner-to-learner, or learner-to-teacher) to encourage active engagement. Levels: Not achieved, Partially achieved, Largely achieved, Fully achieved
  36. 36. Checklist 2: Quality of design Dimension Criteria Level Feedback mechanism Feedback by an academic tutor is limited and scalable (characteristic of a MOOC). The course provides learners with regular feedback through self- assessment activities, tests or peer feedback. The frequency of monitoring has been planned (forum, group, post) A weekly announcement or mass mailing with orientations for the following week is planned. In each weekly session, the pedagogical team makes a synthesis of artefacts from the previous week’s session. Some live events (Hangout, Tweetchat) are scheduled Levels: Not achieved, Partially achieved, Largely achieved, Fully achieved
  37. 37. QRF from MOOC-Quality.eu Phases / perspectives / roles
  38. 38. QRF phases, processes, criteria • Analysis 7 processes 25 criteria • Design 11 processes 54 criteria • Implementation 6 processes 16 criteria • Realization 3 processes 35 criteria • Evaluation 4 processes 11 criteria
  39. 39. Poll – your preference • Few generic benchmarks supplemented by evidence • Many detailed criteria
  40. 40. Learning design
  41. 41. What is learning design? • A way of documenting the design of a course • A way of thinking about & discussing design – before it is too late! • A way to think about appropriate use of technologies • A framework for evaluating courses • A framework for evaluating designs of courses
  42. 42. The 7Cs of Learning Design (Gráinne Conole) Many activities to help teams consider each of these stages – Course tweet (elevator pitch) – Personas – Resource audit – Interactive > Constructive > Active > Passive – Constructive alignment (of learning outcomes, activities, assessment) http://www2.le.ac.uk/projects/oer/oers/beyond-distance-research-alliance/7Cs-toolkit
  43. 43. UCL Learning Designer (Diana Laurillard) • Acquisition • Inquiry • Practice • Production • Discussion • Collaboration https://www.ucl.ac.uk/learning-designer/ Laurillard, D. (2002). Rethinking university teaching: A conversational framework for the effective use of learning technologies. Routledge.
  44. 44. UCL Learning Designer (Diana Laurillard) https://www.ucl.ac.uk/learning-designer/
  45. 45. 8 Learning Events (Leclercq & Poumay) • Imitation / modelling • Reception / transmission • Exercising / feedback • Exploration / resources • Experiment / scenarios • Creation / facilitate • Reflection / advise • Debate / prompting www.labset.net/media/prod/8LEM.pdf
  46. 46. First Principles of Instruction (Merrill) – with a few more! (Margaryan) 1. Problem-centred 2. Activation 3. Demonstration 4. Application 5. Integration 6. Collective knowledge 7. Collaboration 8. Differentiation 9. Authentic resources 10. Feedback Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of massive open online courses (MOOCs). Computers & Education, 80, 77-83.
  47. 47. OU Learning design – module map
  48. 48. OU Learning design – activity classes Assimilative read, watch, listen, think about, observe Finding & handling info list, analyse, collate, find, select, manipulate Communicative communicate, debate, discuss, collaborate, present Productive build, write, make, design, construct, produce, draw Experiential practice, apply, experience, investigate, perform Interactive/adaptive explore, experiment, improve, model, simulate Assessment write, demonstrate, critique, peer review, self-assess, receive feedback
  49. 49. OU Learning design – activity planner
  50. 50. OU Learning design – in practice Toetenel, Lisette and Rienties, Bart (2016). Learning Design – creative design to visualise learning activities. Open Learning, 31(3) pp. 233–244.
  51. 51. OU Learning design – in practice Toetenel, Lisette and Rienties, Bart (2016). Learning Design – creative design to visualise learning activities. Open Learning, 31(3) pp. 233–244.
  52. 52. In summary… • A quality framework should underpin e-learning provision – to help create a quality culture – that is more likely to produce quality e-learning – and quality enhancement • And that also applies to MOOCs • There is no simple recipe, but… – Work in a course team – Think about learning design at an early stage – Don’t let QA procedures get in the way of the day job!
  53. 53. THANK YOU Jon.Rosewell@open.ac.uk
  54. 54. Extra slides!
  55. 55. What students want – and what they need “Student satisfaction is “unrelated” to learning behaviour and academic performance, a study has found. […] while students dislike collaborative learning, they are more likely to pass if they take part in it” (Times Higher Education, Feb 12th 2018) From an analysis of 100,000 students on 151 modules More at Bart Rientes, OU Inaugural Lecture
  56. 56. How does student satisfaction relate to module performance?Satisfaction Students who successfully completed module Slide from Bart Rienties Inaugural lecture
  57. 57. MOOC case study: OU + FutureLearn A representative Open University MOOC … published on FutureLearn • Evidence for OpenupEd features and benchmarks • Quality emerges from joint efforts of OU (university) & FutureLearn (platform provider) • Holistic approach: • Institutional and course level • Process as well as product • Structures and processes embed a concern for quality throughout development, delivery and evaluation Jansen, D., Rosewell, J., & Kear, K. (2017). ‘Quality Frameworks for MOOCs.’ In: M. Jemni, Kinshuk, & M. K. Khribi (Eds.), Open Education: from OERs to MOOCs, 261–281. Springer http://oro.open.ac.uk/47595/
  58. 58. Summary
  59. 59. In summary… • A quality framework should underpin e-learning provision – to help create a quality culture – that is more likely to produce quality e-learning – and quality enhancement • There is no simple recipe, but… – Work in a module team – Think about learning design – Think about student support
  60. 60. Checklist 1: Is it a MOOC or not? Dimension Criteria Level Massive Pedagogical model means effort doesn’t increase significantly as number of participants increase Open Course accessible to (almost) all people without limitation Full course experience available without cost Online All aspects are delivered online Course At least 1 ECTS (25-30 hours of study) Participants receive some feedback (e.g. automated quizzes, peers, general feedback from staff) At least some recognition like badge or certificate of completion. Levels: Not achieved, Partially achieved, Largely achieved, Fully achieved

×