A Systematic Analysis And Synthesis of the Empirical MOOC Literature Published in 2013-2015
•Download as PPTX, PDF•
6 likes•5,180 views
This document summarizes a systematic analysis of 183 empirical research papers on MOOCs published between 2013-2015. The analysis examined the geographic distribution of research, publication outlets, citations, data collection/analysis methods, and research strands. Key findings include: Most research was conducted in the US and published in certain journals/conferences; studies of student topics and MOOC design were most common; quantitative methods like surveys dominated; and there was a lack of instructor-focused and qualitative research. The authors call for more diverse methodological approaches to understanding MOOCs.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to A Systematic Analysis And Synthesis of the Empirical MOOC Literature Published in 2013-2015
Similar to A Systematic Analysis And Synthesis of the Empirical MOOC Literature Published in 2013-2015 (20)
More from Rich McCue
More from Rich McCue (10)
Recently uploaded
Recently uploaded (20)
A Systematic Analysis And Synthesis of the Empirical MOOC Literature Published in 2013-2015
- 1. A Systematic Analysis And Synthesis of the Empirical MOOC Literature Published in 2013-2015 George Veletsianos - Canada Research Chair in Innovative Learning & Technology, Royal Roads University, Peter Shepherdson - University of Zurich, Laura Pasquini - University of North Texas, & Rich McCue - UVic The 9th Collaboration for Online Higher Education & Research (COHERE) Conference October 2015, Victoria & Halifax, Canada
- 2. Rich McCue University of Victoria George Veletsianos, Canada Research Chair in Innovative Learning & Technology, Royal Roads University Peter Shepherdson University of Zurich Laura Pasquini University of North Texas
- 3. • What is the state of MOOC research (very briefly)? • Where are there gaps & what gaps can we fill? • Research Methods. • Did we find anything interesting? Overview
- 4. Introduction We wanted to address a number of gaps in the scholarly understanding of MOOCs and present a comprehensive picture of the MOOC literature by examining: geographic distribution, publication outlets, citations, data collection and analysis methods, and research strands of empirical research focusing on MOOCs.
- 5. Review of the Literature
- 6. cMOOCs and xMOOCs? ● cMOOCs are described as being “based on principles of connectivism, openness, and participatory teaching” (Jacoby, 2014, p. 76), “through a dynamic network of connections afforded by online technology” (Ebben & Murphy, 2014, p. 333). ● xMOOCs resemble “traditional teacher-directed course[s], yet automated, massive, and online” (Kennedy, 2014, p. 8). ● Early MOOCs tended to follow the cMOOC model, whereas more recently the number of xMOOCs delivered has been growing rapidly.
- 7. Expected impacts of MOOCs on education The size and openness of MOOCs have the potential to disrupt conventional thinking about the role, value, and cost of higher education. The discourse around MOOCs could diminish of the authority and importance of the educational leader. MOOCs may threaten the business models of universities.
- 8. Demographics of MOOC users • Vast majority male, 20-40 yrs old. • College degree or higher. • More than 1/2 of learners from outside the United States. • Majority of participants are from North America and Europe, with a small minority being from Asia, Southeast Asia, or Africa.
- 9. Challenges for MOOC’s Completion rates in MOOCs are less than 10%. Why? A lack of incentive, many courses free. Insufficient prior knowledge (e.g., lack of math or language skills). Failure to understand the content and having no one to turn to for help, Lack of time due to having other priorities and commitments. 90%
- 10. Research Questions: RQ1: How is MOOC research geographically distributed? RQ2: Is MOOC research usually published in journal or conference proceedings? In which journals and conference proceedings is MOOC research currently being published? RQ3: Which empirical MOOC studies are cited the most? RQ4: What data collection methods and data analysis methods are used in empirical studies of MOOCs? RQ5: What are the research strands of empirical MOOC research?
- 11. Research Methods
- 12. Literature discovery searches were conducted using the [NEXT] keywords “MOOC” or “Massive Open Online Course.” To be included articles had to be: (1) empirical, (2) published in a peer-reviewed journal, in conference proceedings, or in Educause Review, (3) published or was available online as in press between January 2013 and January 2015, and (4) written in English. All identified papers were examined by [NEXT] two researchers to ensure consensus that they fit the inclusion criteria. Forwarding references: In [NEXT] Google scholar looked up the papers and the looked at papers that cited those papers to discover new papers. Found 60 new papers. Completeness Search: Examined references of 17 papers published in 2015 in our corpus to identify any papers that we missed. They were more likely to reference literature published in 2013-2014 than the papers published in 2013 or 2014. 5 new papers found. [NEXT] The final number of published papers that constituted the corpus of this study was 183. Data Collection Search Keywords: “MOOC” or “Massive Open Online Course” Published Papers: n=183 Authors: n=460
- 13. Limitations 1. Limited duration - 3 years. 2. Only English language data. 3. Quality of research not evaluated. 4. Some content analyzed, but not all.
- 14. Results
- 15. RQ1: How is MOOC research geographically distributed? Country Corpus USA 50.2% UK Kingdom 10% Australia 7.7% China 5.4% Spain 4.8% Canada 4.5% Germany 2.2% Switzerland 1.3% Netherlands 1.1% Other (29 countries) 12.8% Author Affiliations
- 16. RQ2a: Is MOOC research usually published in journals or conference proceedings?
- 17. RQ2b: In which journals and conference proceedings is MOOC research currently being published? Outlet name # of papers Type International Review of Research in Open and Distributed Learning (IRRODL) 18 Journal Proceedings of the first ACM conference on Learning @ scale (L@S '14) 11 Conference Journal of Online Learning and Teaching (JOLT) 7 Journal Conference on Empirical Methods in Natural Language Processing (EMNLP) 5 Conference Distance Education 5 Journal NIPS Workshop on Data Driven Education 5 Conference Proceedings of the 2014 ASCILITE Conference 4 Conference EDUCAUSE Review 4 Journal
- 18. RQ3: Which empirical MOOC studies are cited the most? Paper # of citations - Studying learning in the worldwide classroom research into edX’s first MOOC. 124 - Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. 117 - MOOCs and the funnel of participation. 83 - Tuned Models of Peer Assessment in MOOCs. 73 - Automated Feedback Generation for Introductory Programming Assignments. 56 - Retention and Intention in Massive Open Online Courses: In Depth. 41 - “Turn on, Tune in, Drop out”: Anticipating student dropouts in Massive Open Online Courses. 35
- 19. RQ4: What data collection methods & analysis are used in empirical studies of MOOCs? Analytic method Frequency (%) of total papers Descriptive statistics 93.4 Correlational 52.5 Basic qualitative study 38.8 Experimental and quasi-experimental 25.7 Grounded Theory 7.6 Natural Language processing 7.6 Social Network Analysis 6.6 Ethnography 4.4 Phenomenology 2.2 Discourse analysis 1.0
- 20. RQ5: What are the research strands of empirical MOOC research? ● [NEXT] 83.6% of the papers focused on student-related areas. ● [NEXT] Nearly half (46.4%) of the papers identified in the literature search had some focus on topics relating to the design, creation, and implementation of MOOCs themselves. This included research pertaining to methods of assessment, the description of unique learning environments, the creation of MOOCs on specific topics, and the evaluation of course success. ● [NEXT] Only 8.2% of papers focused on topics related to instructors and teaching. Papers within this theme largely focused on academics’ awareness, perspectives of, and experiences with MOOCs. ● [NEXT] 10.9% of the corpus contained content pertaining to the context and impact of MOOCs. This included research into perceptions of MOOCs, their usefulness as an educational medium, and their economic impact . ● [NEXT] Other. 84% 46% 8% 11%
- 22. Dependence on particular research methods may restrict our understanding of MOOCs Analysis suggests that researchers have favored a quantitative, if not positivist approach to the conduct of MOOC research.
- 23. Survey data and secondary data collected via automated methods were also favored. While some interpretive research was conducted in MOOCs in this time period, it was often basic and only a handful of studies were informed by methods traditionally associated with qualitative research (e.g., interviews, observations, focus groups).
- 24. Thus, even though results suggest that research on MOOCs focuses on student-related topics, learners’ voices are largely absent in the literature. Based on these results, we suggest that an expansion of the methodological approaches used in MOOC research is urgently needed.
- 25. Paucity of research examining instructor-related topics Analysis shows that there is limited research reported on instructor-related topics. This is a rich area for future research. Topics of interest in this area may include instructor motivations, experiences, and perceptions.
- 26. Understanding Learner Subpopulations Completing Auditing Disengaging Sampling No-shows Observers Drop-ins Latecomers Drop-in latecomers Non-engaged Engaged
- 27. Understanding Learner Subpopulations Little research examines the experiences of different populations and how and why learning experiences differ between groups. Future research into this area for instance could examine why some learners disengage, how the learning experience of drop-ins differs from the learning experience of those who are non- engaged, and what interventions may scaffold different types of learners.
- 28. The geography of MOOC research Our geographical analysis of author affiliations showed that over half of the authors conducted their research in the USA. Over 80% of authors were affiliated with institutions in North America or Europe.
- 29. Percentage of Research Published in USA In contrast, according to a Scimago search of all citable documents published in 2013 across all disciplines only 19% came from the USA, and it takes the top 20 countries (including at least seven countries from outside of North America and Europe) to account for 80% of academic output. As such, current MOOC research is simply not a reflection of geographical contributions to academic output in general. 0% 15% 30% 45% 60% MOOC Resarch All Other Research
- 30. Future Directions We hope that this systematic analysis enables researchers to make better sense of the empirical literature on MOOCs and its direction and limitations. There are many possibilities for future research in this area. Future systematic reviews of the literature may focus on: Synthesizing the knowledge on particular areas of interest, (completion and retention in MOOCs; learner motivations in MOOCs). Examining whether research methods used to understand MOOCs follow standard methods of inquiry. Follow methods that take into advantage the digital nature of learning and teaching in this context.
- 31. Final Thoughts Finally, we hope that our results highlight the need for a critical reflection on the part of researchers as to the methods they have used to understand MOOCs to date and encourage a greater diversity in research methods.
- 32. Thank you! Research available at: http://veletsianos/publications This presentation: http://slideshare.net/richmccue Contact: veletsianos@gmail.com @veletsianos on twitter @richmccue on twitter Photo credit: Got Credit
- 33. Special Thanks to the Creators of the Creative Commons Images used in this presentation: • https://www.flickr.com/photos/47572798@N00/8397808475 • https://commons.wikimedia.org/wiki/File:Wet_winding_road.jpg • https://pixabay.com/en/world-map-map-world-black-earth-297446/ • http://creativecommons.org/weblog/entry/26283 • https://en.wikipedia.org/wiki/Printing_press#/media/File:PrintMus_038.jpg • https://commons.wikimedia.org/wiki/File:2011_Library_of_Congress_USA_5466788868_card_catalog.jpg • https://upload.wikimedia.org/wikipedia/commons/7/72/Rhll_wire_rope.jpg • https://upload.wikimedia.org/wikipedia/commons/0/0a/Urval_av_de_bocker_som_har_vunnit_Nordiska_radets_litteraturpris_unde r_de_50_ar_som_priset_funnits_(3).jpg • https://www.flickr.com/photos/mathplourde/8620174342/sizes/l/in/photostream/ • http://orig00.deviantart.net/34c5/f/2011/263/d/6/destruction_by_tbh_1138-d4af5vp.png • https://www.flickr.com/photos/68751915@N05/6355351769 • https://upload.wikimedia.org/wikipedia/commons/1/10/ECLA_of_Bard_Seminar,_2012.jpg • https://www.flickr.com/photos/carbonnyc/143186839 • https://pixabay.com/en/beer-quit-quitting-can-vintage-398742/ • https://pixabay.com/en/blueprint-ruler-architecture-964629/ • https://www.flickr.com/photos/keithroper/8139626676 • http://www.picserver.org/images/highway/phrases/results.jpg • https://upload.wikimedia.org/wikipedia/commons/4/4b/Colossus.jpg • http://www.apa.org/images/apastyle_logo.gif • https://upload.wikimedia.org/wikipedia/commons/3/39/Student_in_Class_(3618969705).jpg • https://upload.wikimedia.org/wikipedia/commons/e/e6/Avrocar_schematic_(high_resolution).jpg • http://teachers.net/teachers.jpg • https://pixabay.com/p-90781/?no_redirect • https://pixabay.com/en/people-sitting-chairs-in-front-690810/ • https://upload.wikimedia.org/wikipedia/commons/1/17/1895-Dictionary-Phrenolog.png • https://www.flickr.com/photos/safari_vacation/10362491406 • https://www.flickr.com/photos/16210667@N02/15617432129 • http://www.veletsianos.com/wp-content/uploads/2013/05/edtechmooc.jpg • https://www.flickr.com/photos/intersectionconsulting/7537238368/ • https://pixabay.com/en/compass-antique-map-of-the-world-429772/ • http://asiimdesgraphic.deviantart.com/art/Mirror-Ball-Reflection-214407739
Editor's Notes
- … I’m going be reviewing the highlights of our paper titled, “A Systematic Analysis And Synthesis of the Empirical MOOC Literature Published in 2013-2015”
- I’m presenting on behalf of our Research Team: George Veletsianos Peter Shepherdson Laura Pasquini Rich McCue
- Overview: What is the state of MOOC research (I’ll keep this part very brief)? Where are there gaps & what gaps can we fill? I’ll go over Research Methods. Did we find anything interesting? ----> I won’t keep you in suspense, [NEXT] YES we did discover some interesting findings.
- A deluge of empirical research became available on MOOCs between 2013-2015. The goal of our research was to address a number of gaps in the scholarly understanding of MOOCs and present a comprehensive picture of the MOOC literature by examining the: [NEXT] geographic distribution, [NEXT] publication outlets, [NEXT] citations, [NEXT] data collection and analysis methods, and [NEXT] strands of empirical research focusing on MOOCs.
- Review of the Literature
- What difference is there between cMOOC’s and xMOOC’s: cMOOCs are described as being “based on principles of connectivism, openness, and participatory teaching” (Jacoby, 2014, p. 76). xMOOCs resemble “traditional teacher-directed course[s], yet automated, massive, and online” (Kennedy, 2014, p. 8). Early MOOCs tended to follow the cMOOC model, whereas more recently the number of xMOOCs delivered has been growing rapidly.
- What are some of the expected impacts of MOOC’s on Education? [NEXT] The size and openness of MOOCs have the potential to disrupt conventional thinking about the role, value, and cost of higher education. [NEXT] The discourse around MOOCs could diminish of the authority and importance of the educational leader. [NEXT] MOOCs may threaten the business models of universities.
- What are the general demographics of MOOC users? The vast majority are male, between the ages of 20 and 40, and have already earned a college degree or higher. More than half of learners in MOOCs are from countries other than the United States. A large majority of participants are from North America and Europe, with a small minority being from Asia, Southeast Asia, or Africa.
- What is one of the major challenges for MOOC’s that you’ve probably heard about? [NEXT] Completion rates in MOOCs are less than 10%. And why is that? Some reasons we found in the literature are: A lack of incentives; for example, many courses free. Insufficient prior knowledge (e.g., lack of math or language skills). Failure to understand the content and having no one to turn to for help, Lack of time due to having other priorities and commitments.
- Here are the research questions we addressed in our study, and I’ll go over each one with our associated findings in just a minute...
- But first, Research Methods
- Literature discovery searches were conducted using the [NEXT] keywords “MOOC” or “Massive Open Online Course.” To be included articles had to be: (1) empirical, (2) published in a peer-reviewed journal, in conference proceedings, or in Educause Review, (3) published or was available online as in press between January 2013 and January 2015, and (4) written in English. All identified papers were examined by two researchers to ensure consensus that they fit the inclusion criteria. In [NEXT] Google scholar we looked up each of the identified papers and then looked at all papers that cited the identified paper in order to discover new papers. We found 60 new papers and included them in our corpus. In addition, we examined references of 17 papers published in 2015 to identify any papers that we might have missed. They were more likely to reference literature published in 2013-2014 than the papers published in 2013 or 2014. 5 new papers found. [NEXT] The final number of published papers that constituted the corpus of this study was 183 with [NEXT] 460 Authors.
- This study clarifies the state of the literature published at a particular point in time, and there are three limitations arising from this research context. First, this study draws upon less than three years of English Language data and its findings are only representative of the research on MOOCs at an early stage. Second, the data analysis methods used do not allow us to judge the quality of the research reported. The papers included in our corpus are of mixed quality. For instance, our reporting on the use of grounded theory does not necessarily examine whether the authors used the method correctly, rigorously, or even uniformly. Third, while our data reflect some of the content of the papers analyzed, they do not reflect a full evaluation of the contents of the papers.
- Results
- RQ1: How is MOOC research geographically distributed? We determined the geographic distribution of research papers by looking at the home institution of the lead author on each paper, and then categorized the institution by country and region. We did the same for each for each of the 460 authors. The vast majority of authors came from North America and Europe, which between them accounted for over 82% of the author affiliations [NEXT] More than half of the authors were affiliated with institutions from the USA. Top 9 countries represented 87% of the author affiliations. The other 13% of authors represented 29 other countries.
- RQ2a: Is MOOC research usually published in journals or conference proceedings? 98 papers were published in peer-reviewed journals and 85 were published in conference proceedings.
- RQ2b: In which journals and conference proceedings is MOOC research currently being published? The top three are: International Review of Research in Open and Distributed Learning. Proceedings of the first ACM conference on Learning @ scale. Journal of Online Learning and Teaching (JOLT).
- RQ3: Which empirical MOOC studies are cited the most? We determined citation frequency by In looking up each article in Google scholar and used the Google Scholar “Cited By” number. I’ll let you read the study titles, but my favourite title on the list is: “Turn on, Tune in, Drop out”: Anticipating student dropouts in Massive Open Online Courses. I suspect I like this title, because of the embarrassingly large number of MOOC’s I’ve signed up for, but by the time the start date rolled arounds, discovered I was too busy to add a course to my schedule. Of the 183 papers identified, almost half (48%) were cited 0 times, and Seventy-two papers were cited one to ten times. Thirteen papers were cited 25 or more times.
- RQ4: What data collection methods and data analysis methods are used in empirical studies of MOOCs? This data was gathered by two researchers who analysed each article for data collection and analysis methods… (aside: which gave one researcher in particular an appreciation for well structured and formatted journal articles) ;-) For starters 45% of the papers used one data collection method and 38% used two data collection methods. Automated collection of secondary data (e.g., trace data from Learning Mgt Systems) was used 73% of the time. The second most popular data collection method was questionnaires/surveys, which were used in 56% of papers. The rest of the data collection methods were used much less frequently. Automated methods of data collection were used as the sole data collection method in 27% of the corpus. Questionnaires and surveys were used as the sole data collection method 10% of the time.
- RQ5: What are the research strands of empirical MOOC research? This data was gathered by two researchers who analyzed each article for research strands. [NEXT] 84% of the papers focused on student-related areas, including research on learner behaviors, motivation, performance, participation, preferences, and interaction. Given the attention that MOOC completion rates have received in the mass media, it is perhaps unsurprising that numerous researchers have examined completion, retention, and learner subpopulations. For instance, low completion rates (often less than 10% of registrants), vary substantially among learners with different intentions. [NEXT] Nearly half, or 46% of the papers had some focus on topics relating to the design, creation, and implementation of MOOCs themselves. This included research pertaining to methods of assessment, the description of unique learning environments, the creation of MOOCs on specific topics, and the evaluation of course success. Common within this theme was research that investigated the utility of individual elements of MOOCs. For example, some researchers focused on the inclusion of tools for social interaction within online courses, while others investigated the use of specific types of media in instruction. There was also a strong focus on means of assessing student work [NEXT] Only 8% of papers focused on topics related to instructors and teaching. Papers within this theme largely focused on academics’ awareness, perspectives of, and experiences with MOOCs. [NEXT] 11% of the corpus contained content pertaining to the context and impact of MOOCs. This included research into perceptions of MOOCs, their usefulness as an educational medium, and their economic impact.
- Discussion & Conclusions
- Our analysis suggests that researchers have strongly favored a quantitative, if not positivist approach to the conduct of MOOC research. This dependence on these particular research methods may restrict our understanding of MOOCs I personally suspect that part of the reason the quantitative research approach is used so frequently, is the relative ease of accessing clickstream data from MOOC environments, as opposed to the difficulties and expense involved in conducting many forms of qualitative research. “Positivism is a philosophical theory stating that positive knowledge is based on natural phenomena and their properties and relations. Thus, information derived from sensory experience, interpreted through reason and logic, forms the exclusive source of all authoritative knowledge.” - https://en.wikipedia.org/wiki/Positivism
- Not surprisingly, survey data and secondary data collected via automated methods were also favored. While some interpretive research was conducted in MOOCs in this time period, it was often basic and only a handful of studies were informed by methods traditionally associated with qualitative research (e.g., interviews, observations, focus groups).
- Thus, even though our results suggest that research on MOOCs focuses on student-related topics, learners’ voices are largely absent in the literature. These results provide empirical support to the claim published by Veletsianos, Collier, and Schneider that “the MOOC phenomenon experienced a surge of research using quantitative, clickstream and observational data” and suggest that what we know about MOOCs may be the result of the field’s overwhelming dependence on particular data collection and analysis methods. Based on these results, we suggest that an expansion of the methodological approaches used in MOOC research is urgently needed. Given that research into MOOCs is expected to inform learning in all environments and not just MOOCs, a broader methodological toolkit is imperative.
- Analysis shows that there is limited research reported on instructor-related topics. This is a rich area for future research. Topics of interest in this area may include: instructor motivations, experiences, and perceptions. Researchers could examine how instructors experience the design and development of these courses, why they choose to teach MOOCs, and how they perceive their relationship with MOOC learners (and whether that relationship differs from traditional student-learner relationships). Given that a number of MOOCs enlist the help of instructional assistants (e.g., Teaching Assistants and course “alumni”), research in this area could investigate the impact of instructional assistants on learning and support, as well as the experiences that instructional staff have in the delivery of the course.
- Results show that a number of researchers have attempted to identify and classify learners into various groupings. For instance, the literature suggests that MOOC learners can be described as completing, auditing, disengaging, sampling, and so on...
- Little research examines the experiences of different populations and how and why learning experiences differ between groups. One exception is Huang et al who examined the quality and impact of discussion forum posts made by high-volume contributors. Future research into this area for instance could examine: why some learners disengage, how the learning experience of drop-ins differs from the learning experience of those who are non-engaged, and what interventions may scaffold different types of learners.
- Our geographical analysis of author affiliations showed that over half of the authors conducted their research in the USA. Over 80% of authors were affiliated with institutions in North America or Europe.
- In contrast, according to a “Scimago" search of all citable documents published in 2013 across all disciplines only 19% came from the USA, and it takes the top 20 countries (including at least seven countries from outside of North America and Europe) to account for 80% of academic output. As such, current MOOC research is simply not a reflection of geographical contributions to academic output in general.
- We hope that our systematic analysis enables researchers to make better sense of the empirical literature on MOOCs and its direction and limitations. There are many possibilities for future research in this area. Future systematic reviews of the literature may focus on: Synthesizing the knowledge on particular areas of interest, (e.g. completion and retention in MOOCs; learner motivations in MOOCs). Examining whether research methods used to understand MOOCs follow standard methods of inquiry. Use methods that take advantage of the digital nature of learning and teaching in this context.
- Finally, we hope that our results highlight the need for a critical reflection on the part of researchers as to the methods they have used to understand MOOCs to date and encourage a greater diversity in research methods.