The document discusses the evaluation of MOOC accessibility and quality through various studies, interviews, and audits conducted among providers and disabled learners. It highlights the necessity for better quality in MOOCs to enhance accessibility for all learners, regardless of disabilities, while addressing barriers that still exist in providing formal recognition and pathways for academic qualifications. Several quality frameworks and criteria for evaluating MOOCs are proposed, along with an audit checklist to measure accessibility effectiveness.

1. Checking MOOC quality afterwards:
the case of accessibility
Francisco Iniesto
The Open University
Webinar: Quality frameworks for MOOCs
18 September 2018

2. Study A: The perspectives of
providers
Study C: MOOC accessibility
audit
Study B: The perspectives of
learners
26 Interviews with
MOOC providers and
those with a range of
expertise in the MOOC
community
15 interviews with
disabled learners
Pre and post MOOC
survey data from 14
Open University MOOCs
at FutureLearn
1. Accessibility heuristic
evaluation
2. UX walkthroughs and
design in learning
3. Quality of the design
4. Universal design for
Learning (UDL)
Being tested on
FutureLearn, edX,
Coursera and Canvas.
Contextualisation
An Investigation into the accessibility of MOOCs (Iniesto et al., 2018)

3. The MOOC accessibility audit

4. Quality evaluation
Accessibility vs quality?
“The increase of quality in MOOCs is going to help a better accessibility to all learners,
disregarding of disabilities.
1. Mulder and Jansen (2015) conclude, when exploring if MOOCs can be an instrument in
opening up education, that MOOCs cannot remove all barriers to learning but can
contribute to ensuring quality education to all.
2. In the same line of argumentation, Schuwer et al. (2015) point out the lack of MOOCs to
offer formal pathways to recognise academic qualifications and the inequality of
access that provides.
3. Stracke et al. (2018) reflect on the need to provide MOOCs with better quality to
address new target groups, to be used in multicultural and multilingual environments.“

5. Quality evaluation
Background:
1. Quality Frameworks for MOOCs (Jansen et al., 2016)
2. OpenupEd label (Rosewell & Jansen, 2014), based around the E-xcellence
3. ECO eLearning (Osuna et al., 2016), Score 2020 (Score2020, 2016) and BizMOOC
(BizMOOC, 2017) projects
The checklist had to be updated in with the following aspects:
1. The language used had to be changed and oriented to evaluate a MOOC that was already
produced and being run.
2. Select the criteria that are applicable considering a MOOC that is online and is being
evaluated after its production.

6. The MOOC accessibility audit
1. All the checklist share the same structure for every single criterion:
• What to test for: information to help the evaluator to know what the criterion is evaluating.
• Testing method: information to help the evaluator to proceed to test the criterion.
• Comments: space for the evaluator to add free comments.
2. All the checklist share the same rating method:
• NA (Not achieved): The feature to test is missing.
• PA (Partially achieved): The feature to test is available but not integrated.
• LA (Largely achieved): The feature to test is available and partially integrated.
• FA (Fully achieved): The feature to test is available and fully integrated.
• If the criterion is not relevant, “Not Applicable” is added to the comments

7. Quality evaluation
Criteria Reason to be removed
Accessibility Redundancy with the rest of the checklists of the audit
Visibility Focused internally on the organisation
Support to staff Focused internally on the organisation
Criteria Decision taken
Massive To remain in the checklist
Course To remain in the checklist
Target group Partially removed from the checklist
Learning Objectives / Outcomes To remain in the checklist as learning objectives
Content / Assets Partially removed from the checklist as educational resources
Feedback mechanism To remain in the checklist
Platform To remain in the checklist
Support for MOOC participants To remain in the checklist as support for MOOC learners
Study unit European Credit Transfer and Accumulation System
(ECTS) updated into number of hours
Group of criteria to be removed
Group of criteria that needed an extra explanation

8. Quality evaluation
Group of criteria Criteria Number Total
The MOOC criteria Massive 1 13
Open 5
Online 1
Course – study unit 1
Full course 5
Quality of the design Target group 2 28
Workload 2
Overall goal 1
Learning objectives 5
Learning activities 5
Educational
resources
5
Feedback
mechanism
6
Assessment 4
Technical platform
and support for
learners
Platform 2 9
Support for MOOC
learners
7
50
1.2.1 Open - enrolment
1.2.2 Open - access
1.2.3 Open - location
1.2.4 Open - prior knowledge
1.2.5 Open - free

9. Current work
✓ 1st run of the accessibility audit through FutureLearn, Coursera, edX and Canvas
✓ Analyse findings
✓ Optimisation of the audit
✓ Overlap between the four parts of the audit, unification of criteria

10. References
• BizMOOC. Report 5.3.Quality Assurance Report 2 (2017)
• Iniesto, F., McAndrew, P., Minocha, S., & Coughlan, T. (2017). An investigation into the perspectives of
providers and learners on MOOC accessibility. In: TEEM'17: international conference technological
ecosystems for enhancing multiculurality
• Jansen, D., Rosewell, J., & Kear, K. (2016). Quality Frameworks for MOOCs. In M. Jemni, Kinshuk & M.K.
Khribi (Eds.), Open Education: from OERs to MOOCs. Lecture Notes in Educational Technology (LNET). pp.
261-281 Aug 18, 2016. Berlin: Springer.
• Osuna Acedo, S., Frau-Meigs, D., Camarero Cano, L., Bossu, A., Pedrosa, R. & Jansen, D. (2016).
Intercreativity and interculturality in the virtual learning environments of the ECO MOOC project. In M.
Jemni, Kinshuk & M.K. Khribi (Eds.), Open Education: From OERs to MOOCs. Berlin: Springer.
• Mulder, F., & Jansen. D. (2015). MOOCs for Opening Up Education and the OpenupEd initiative. In: C. J.
Bonk, M. M. Lee, T. C. Reeves, T. H. Reynolds (Eds.). The MOOCs and Open Education Around the World. New
York: Routledge Tayler & Francis Group.
• Rosewell, J., & Jansen, D. (2014). The OpenupEd quality label: benchmarks for MOOCs. INNOQUAL: The
International Journal for Innovation and Quality in Learning, 2(3), 88–100.
• Schuwer, R., Gil-Jaurena, I., Hakan Aydin, C., Costello, E, Dalsgaard, C., & Brown, M., et al. (2015).
Opportunities and threats of the MOOC movement for higher education: the European perspective.
International Review of Research in Open and Distributed Learning, Special Issue on European MOOCs. 16
(6), 20–38. ISSN 1492-3831.
• Stracke, C. M., Tan, E., Texeira, A. M., Pinto, M., Kameas, A., Vassiliadis, B., & Sgouropoulou, C. (2018). Gap
between MOOC designers’ and MOOC learners’ perspectives on interaction and experiences in MOOCs:
Findings from the Global MOOC Quality Survey. Proceedings 18th IEEE International Conference on
Advanced Learning Technologies (ICALT) (pp. 1-5). IEEE: Computer Society.

11. Checking MOOC quality afterwards: the
case of accessibility
Francisco Iniesto
The Open University
Francisco.Iniesto@open.ac.uk