Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2018-08-23 EARLI Conference in Bonn Quality Reference Framework for MOOCs Stracke Tan

118 views

Published on

2018-08-23 Paper Presentation at EARLI SIG 6-7 Conference in Bonn on The Quality Reference Framework for MOOCs by Christian M. Stracke and Esther Tan from OUNL

Published in: Education
  • Be the first to comment

  • Be the first to like this

2018-08-23 EARLI Conference in Bonn Quality Reference Framework for MOOCs Stracke Tan

  1. 1. Towards a Quality Reference Framework for MOOCs @ EARLI SIG 7 in Bonn, 2018-08-23 by Christian M. Stracke & Esther Tan Open University of the Netherlands
  2. 2. Open CC License for sharing & re-using slides This work is free to share under the creative commons licence: "Attribution – Noncommercial – Share Alike 3.0" You can copy, distribute and transmit the work under the following conditions: 1. Attribution – 2. Noncommercial – 3. Share Alike Licence: Attribution-Noncommercial-Share Alike Some rights reserved, see: http://creativecommons.org/licenses/by-nc-sa/3.0/
  3. 3. Open University of the Netherlands Global cooperation: ECNU & KNOU Global initiative ICORE for OR & OE International WLS / LINQ Conference eLC European Institute ICDE Chair in OER Dr. Christian M. Stracke: Open Learning & Education, Innovations, Policies, Quality & Competences, Impact
  4. 4. Open University of the Netherlands The Quality of MOOCs Let’s Learn to Learn Seamless Learning Dr. Esther Tan Technology-Enhanced Learning, Innovations in & out Classroom
  5. 5. MOOQ and the Quality of MOOCs: What are we talking about?
  6. 6. Two core factors: 1. Globalisation & 2. Worldwide Internet The Digital Age Photo: https://ec.europa.eu/digital-agenda/en/digital-life/education
  7. 7. Global Competitions and societal changes Close the gaps & open new opportunities Challenges: Learn to Learn Internationalization Figure: http://www.weforum.org/issues/global-competitiveness
  8. 8. We need Change in Education!
  9. 9. To improve Quality in Education!
  10. 10. Open Education (OE): Innovations for changing & opening up education to improve the quality
  11. 11. Goal 4: Inclusive and quality education Sustainable dev goals www.sustainabledevelopmentgoals.org
  12. 12. Goal 4: Inclusive and quality education Sustainable dev goals
  13. 13. What is Quality Education?
  14. 14. What is Quality?
  15. 15. A simple experiment ...
  16. 16. A simple experiment ...
  17. 17. Quality cannot be defined ... except through adaptation to your situation and context!
  18. 18. Quality Standard with Reference Process Model: Standard:ISO/IEC 40180 Communication concept Needs Analysis Conception / Design Development / Production Implementation Framework Analysis Learning Process / Realization Initiation Media realization Technical realization Design realization Content realizationAnalysis of the external context Testing of learning resources Learning objectives Stakeholder identification Concept for contents Definition of objectives Demand analysis Time and budget planning Environment analysis Activities Organization of use Activation of learning resources Adaptation of learning resourcesDidactical concept/ methods Organizational concept Roles and activities Technical concept Concept for media and interaction design Media Concept Technical infrastructure Review of competencies levels Concept for tests and evaluation Analysis of staff resources Evaluation / Optimization Planning Realization Analysis Optimization / improvement Analysis of the institutional and organizational context Analysis of target groups Concept for maintenance Administration Maintenance
  19. 19. Example of required adaptation: Standard:ISO/IEC 40180 Communication concept Needs Analysis Conception / Design Development / Production Implementation Framework Analysis Learning Process / Realization Initiation Media realization Technical realization Design realization Content realizationAnalysis of the external context Testing of learning resources Learning objectives Stakeholder identification Concept for contents Definition of objectives Demand analysis Time and budget planning Environment analysis Activities Organization of use Activation of learning resources Adaptation of learning resources Didactical concept/ methods Organizational concept Roles and activities Technical concept Concept for media and interaction design Media Concept Technical infrastructure Review of competencies levels Concept for tests and evaluation Analysis of staff resources Evaluation / Optimization Planning Realization Analysis Optimization / improvement Analysis of the institutional and organizational context Analysis of target groups Concept for maintenance Administration Maintenance
  20. 20. MOOQ and the Quality of MOOCs: What are we talking about?
  21. 21. What are MOOCs?
  22. 22. Source: https://www.flickr.com/photos/catspyjamasnz/ Source: https://www.flickr.com/photos/gforsythe/ What are MOOCs? Massive (?) Open (?) Online (?) Courses (?)
  23. 23. Source: https://www.flickr.com/photos/-ed/ And the Quality of MOOCs? Many issues are leading to high drop-out rates (>90%)
  24. 24. What is the Future of MOOCs? MOOCs are dead! (Stracke, 2017 - and long live the MOOCs!) MOOCs are the educational buzzword of 2012 (Daniel, 2012)
  25. 25. What is the Future of MOOCs? The fact is: Number of MOOCs, MOOC learners and MOOC providers are growing until today (Class Central, 2018)
  26. 26. What is ?
  27. 27. MOOQ for the quality of MOOCs: “We will make MOOCs better” Quality Reference Framework with indicators for design & comparison www.MOOC-quality.eu Frameworks: MOOQ
  28. 28. Coordinator & QRF development lead ENS UAb NQIS HOU OUNL OUNL HOU NQIS UAb ENS MOOC development & QRF contributor Standardization & QRF contributor Research & QRF contributor Research & QRF contributor
  29. 29. Mixed Method Research with many sources & externals RESEARCH QRF DESIGN PILOTING STANDARD STEP 4 STEP 3 2 STEP 1 STEP Development and refinement of QRF with practitioners & experts worldwide Development of two MOOCs for testing the QRF Proposal for QRF as European CEN & international ISO standard
  30. 30. Quality Reference Framework with criteria & checklist for MOOC design Our main goal is the collaboration with all to improve Open Education & MOOCs www.MOOC-quality.eu
  31. 31.  Subjects of Investigation: Learners, Designers, Facilitators & Providers  Mixed Methods Approach: Quantitative & qualitative data 1. Global MOOC Quality Survey, 2. Open-Ended questions (OEQ) & 3. Semi-structured interviews Methodological Framework
  32. 32. Instruments of Measure MOOC Learners MOOC Designers MOOC Facilitators MOOC Providers TOTAL Global MOOC Quality Survey 166 (69 qns.) 68 (89 qns.) 33 (58 qns.) - 267 Open- ended Questions 118 (4 qns.) 42 (4 qns.) 27 (4 qns.) - 185 Semi- structured interviews - 12 (15 qns.) 12 (10 qns.) 12 (13 qns.) 36
  33. 33. MOOC Survey Constructs Constructs Learners Designers Facilitators Experience with MOOC X X X Learning Objectives X X X Duration and Structure X X Learning Resources X X X Accessibility and Inclusion X X Learning Progress X Learning Environment X X Learning Assessment X X X Learning Certification X X Design Process X Pedagogical Decisions X Learning Support  Feedback & Facilitation  Interaction & Collaboration X X X
  34. 34. Global MOOC Quality Survey (GMQS)
  35. 35. Global MOOC Quality Survey
  36. 36. Demographic Profile Age range of all survey participants (learners, designers & facilitators) by gender
  37. 37. Demographic Profile Educational level of all survey participants (learners, designers & facilitators) by gender
  38. 38. n VB B N G VG Learning experience 166 4 4 13 75 70 Learning Experience (Learners)
  39. 39. n VB B N G VG Design experience 68 1 2 13 33 19 Design Experience (Designers)
  40. 40. Interaction from Learners‘ Perspective n N/A SD D N A SA LF 146 20 5 13 48 37 23 LL 146 15 3 17 34 51 26 LR 146 9 2 8 25 61 41 GG 146 37 4 15 50 24 16 Note: LF: Interaction between learners and facilitators LL: Interaction among learners LR: Interaction between learners and learning resources GG: Interaction among teams and groups
  41. 41. n R2 M2 p LF by learners 125 .094 9.382 .000*** LL by learners 130 .101 10.818 .000*** LR by learners 136 .112 12.286 .000*** GG by learners 108 .045 4.131 .026* Bivariate Correlations between LLR4 and LLE4
  42. 42. Interaction from Designers‘ Perspective n N/A SD D N A SA LF 52 2 1 5 11 24 9 LL 52 1 1 3 11 19 17 LR 52 3 1 0 4 22 22 GG 52 8 2 10 14 13 5 Note: LF: Interaction between learners and facilitators LL: Interaction among learners LR: Interaction between learners and learning resources GG: Interaction among teams and groups
  43. 43. n R2 M2 p LF by designers 49 .003 0.109 .703 LL by designers 50 .043 1.595 .143 LR by designers 48 .046 1.537 .138 GG by designers 43 .001 0.038 .821 Bivariate Correlations between DLR4 and DDE4
  44. 44. Open-ended Questions (Learners)
  45. 45. No of respondents across the six domains Domain Learners Designers Facilitators Social Sciences, Humanities & Law 24 9 2 Education & Lifelong Learning 19 12 18 Computing & Informatics 18 5 2 Science, Math & Engineering 16 5 1 Nature, Environment & Health 21 6 1 Business, Management & Economics 20 5 3 Total 118 42 27
  46. 46. MOOC Open-ended Questions (OEQ) MOOC Learners Q1. What were the three main strengths of the MOOC? Q2. What were the three main weaknesses of the MOOC? Q3. What was missing in the MOOC? Q4. Looking ahead into the development of this type of learning experiences, what could be improved in future MOOCs?
  47. 47. Open-ended Questions (MOOC Learners) Q1. What were the three main strengths of the MOOC? 0 10 20 30 40 50 60 70 80 90 Curriculum Design & Delivery Instructional Design & Technology Assessment & Evaluation Facilitation & Feedback Interaction & Collaboration NumberofComments Social Sciences, Humanities and Law Education and Lifelong Learning Computing and Informatics Science, Maths and Engineering Nature, Environment and Health Business, Management and Economics
  48. 48. Open-ended Questions (MOOC Learners) Q2. What were the three main weaknesses of the MOOC? 0 5 10 15 20 25 30 35 Curriculum Design Instructional Design & Technology Assessment & Evaluation Faciliation & Feedback Interaction & Collaboration NumberofComments Social Sciences, Humanities and Law Education and Lifelong Learning Computing and Informatics Science, Maths and Engineering Nature, Environment and Health Business, Management and Economics
  49. 49. Open-ended Questions (MOOC Learners) Three main strengths and weaknesses of the MOOC Strengths Weaknesses Curriculum Design & Delivery • Good choice & quality of content • Good teachers, presenters & tutors • LO aligns with content • Weak choice & quality of content • Short duration Instructional Design & Technology • Good integration of IT & media • Support self-regulation & individual learning paths • Poor use of IT technological tools • Resources lack variety & quality Interaction & Collaboration • Encourage local group discussion & activities • Foster interaction with field experts • Lack interaction: learner-tutor • No support for community building
  50. 50. Open-ended Questions (Designers)
  51. 51. MOOC Open-ended Questions (OEQ) MOOC Designers Q1. Which were the main design decisions that you made during the development of the MOOC that later proved to be successful? Q2. Which were the three biggest difficulties that you faced when designing the MOOC? Q3. Which design decisions did not pay off as you expected? Q4. Looking ahead into the development of this type of learning design experiences, what methods and tools could be helpful to improve the design of future MOOCs?
  52. 52. Open-ended Questions (MOOC Designers) Q1. Which were the main design decisions that you made during the development of the MOOC that later proved to be successful? 1 3 3 42 1 1 3 3 6 5 4 0 4 8 12 16 20 24 Assessibility& Inclusion Assessment & Evaluation Interaction & Collaboration Certification& Accreditation Curriculum Design& Delivery Expertise & Manpower Feedback & Facilitation Instructional Design& Technology No.ofComments Social Sciences, Humanities and Law Education and Lifelong Learning Computer and Informatics Science, Maths and Engineering Nature, Environment and Health Business, Management and Economics
  53. 53. Open-ended Questions (MOOC Designers) Q2. Which were the three biggest difficulties that you faced when designing the MOOC? 3 3 1 3 3 2 8 1 2 2 1 1 1 2 3 5 4 4 2 5 3 0 4 8 12 16 20 24 Assessment & Evaluation Curriculum Design& Delivery Expertise & Manpower Feedback & Facilitation Institutional Support & Funding Instructional Design& Technology Interaction & Collaboration Open Access, Copyrights & Licensing No.ofComments Social Sciences, Humanities and Law Education and Lifelong Learning Computer and Informatics Science, Maths and Engineering Nature, Environment and Health Business, Management and Economics
  54. 54. Open-ended Questions (MOOC Designers) Successful Decisions Biggest Challenges Curriculum Design & Delivery • Content delivery format • Content structure & LOs • Weak choice & quality of content • Short duration Instructional Design & Technology • Choice of learning activities • Integration of IT & media • Platform, software & production decisions Interaction & Collaboration Expertise & Manpower • Creating community of learners • Foster interaction between learner & tutor/ facilitator • Gap in content & instructional design knowledge • Coordination & collaboration, e.g., different experts & teaching staff Three main successful decisions and three biggest challenges
  55. 55. Quality Reference Framework (QRF) for MOOCs
  56. 56. The Quality Reference Framework Dimension 1: Phases Analysis, Design, Implementation, Realization, Evaluation Dimension 2: Perspectives Pedagogical, Technological, and Strategic Dimension 3: Roles Designer, Facilitator, and Provider
  57. 57. Quality Reference Framework (QRF) for MOOCs
  58. 58. Quality Reference Framework (QRF) for MOOCs
  59. 59. Analysis of the needs and demands for the MOOC DESIGN REALIZATION ANALYSIS IMPLEMENTATION EVALUATION Design decisions on all aspects of the MOOC Development of the MOOC following the design Running of the MOOC with all learning and facilitation processes Parallel ongoing evaluation of all other four phases The Quality Reference Framework
  60. 60. MOOQ project: www.MOOC-quality.eu Online community for the Quality Reference Framework: www.MOOC-quality.net
  61. 61. First References for GMQS Stracke, C. M., et al. (2018). Gap between MOOC designers' and MOOC learners' perspectives on interaction and experiences in MOOCs: Findings from the Global MOOC Quality Survey. In M. Chang, N.-S. Chen, R. Huang, Kinshuk, K. Moudgalya, S. Murthy, & D. G. Sampson (Eds.), Proceedings 18th IEEE International Conference on Advanced Learning Technologies (ICALT) (pp. 1-5). IEEE: Computer Society. DOI 10.1109/ICALT.2018.0000 Stracke, C. M., & Tan, E. (2018). The Quality of Open Online Learning and Education: Towards a Quality Reference Framework for MOOCs. In J. Kay, & R. Luckin (Eds.), Rethinking learning in the digital age. Making the Learning Sciences Count: The International Conference of the Learning Sciences (ICLS) 2018 (pp. 1029-1032). London: ISLS. Stracke, C. M. et al. (2017). The Quality of Open Online Education: Towards a Reference Framework for MOOCs. In Proceedings of 2017 IEEE Global Engineering Education Conference (EDUCON) (pp. 1712-1715). IEEE Xplore. DOI: 10.1109/EDUCON.2017.7943080 To be continued …
  62. 62. christian.stracke@ou.nl esther.tan@ou.nl @ChrMStracke @Taneste www.opening-up.education Let’s collaborate!
  63. 63. Thank you! Questions?

×