LAK'17 Conference paper presentation: Abstract: In 2011, the phenomenon of MOOCs had swept the world of education and put online education in the focus of the public discourse around the world. Although researchers were excited with the vast amounts of MOOC data being collected, the benefits of this data did not stand to the expectations due to several challenges. The analyses of MOOC data are very time-consuming and labor-intensive, and require a highly advanced set of technical skills, often not available to the education researchers. Because of this MOOC data analyses are rarely done before the courses end, limiting the potential of data to impact the student learning outcomes and experience. In this paper we introduce MOOCito (MOOC intervention tool), a user-friendly software platform for the analysis of MOOC data, that focuses on conducting data-informed instructional interventions and course experimentations. We cover important design principles behind MOOCito and provide an overview of the trends in MOOC research leading to its development. Although a work-in-progress, in this paper, we outline the prototype of MOOCito and the results of a user evaluation study that focused on system’s perceived usability and ease-of-use. The results of the study are discussed, as well as their practical implications.

1. DEVELOPING A MOOC EXPERIMENTATION
PLATFORM: INSIGHTS FROM A USER STUDY
The University of Edinburgh
Vitomir Kovanović, Srećko Joksimović, Philip Katerinopoulos,
Charalampos Michail, George Siemens, & Dragan Gašević
Simon Fraser University,
Vancouver, Canada
Lak17
@vkovanovic
v.kovanovic@ed.ac.uk
http://vitomir.kovanovic.info

2. WHY ARE MOOCS INTERESTING (FOR A RESEARCHER)
MOOCs
2
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
1. Educational researchers historically small scale – “Big Data” in education
2. Potential for improving on-campus learning experience
3. Innovate for teaching innovations (e.g., flipped classroom models)
4. Great potential for understanding human learning
5. Experiment with different instructional interventions

3. WHY ARE MOOCS INTERESTING (FOR A RESEARCHER)
MOOCs
3
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
1. Educational researchers historically small scale – “Big Data” in education
2. Potential for improving on-campus learning experience
3. Innovate for teaching innovations (e.g., flipped classroom models)
4. Great potential for understanding human learning
5. Experiment with different instructional interventions
ALSO (of course)
1. Democratization of learning
2. Helping students without access to good quality education
3. Closing the achievement gap
4. Helping developing countries
5. Reduce costs of education
6. …..

4. CURRENT RESEARCH CHALLENGES
MOOCs
4
Lak17Kovanovic et al. – Developing a MOOC experimentation platform

5. CURRENT RESEARCH CHALLENGES
MOOCs
5
Lak17Kovanovic et al. – Developing a MOOC experimentation platform

6. CURRENT RESEARCH CHALLENGES
MOOCs
6
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
Data is notoriously had to collect and analyze
1. Diverse data sources
2. Tracking users across platforms (e.g., twitter, surveys)
3. Different data formats (CSV formats, JSON formats)
4. Disconnected data sources (_general, _mappings, _forum)
5. Anonymization
6. IRBs
7. Data analysis requires understanding of the data science methods (e.g., feature engineering)
8. Understanding of the pedagogical domain
9. Correctly interpreting the data
10. Issues with replication of educational studies
11. No ad-hoc experimentation, A/B testing must be planned far in advance

7. CURRENT RESEARCH CHALLENGES
MOOCs
7
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
1. Most MOOC instructors lack these skills
1. One reason for why Computer Science researches dominated MOOC research

8. GOAL
MOOCs
8
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
Develop a tool that will assist MOOC instructors/researchers in analyzing the MOOC data
Guiding principles:
1. No technical prerequisites
1. No database installs
2. Incremental data import
1. As the new data dump becomes available, it should be possible to import it.
3. Support cluster analysis
1. We will add additional types of analysis over time (e.g., SNA)
4. Enable follow-up analysis
1. Details about interventions should be possible to analyze in R/SPSS
5. Support class intervention
1. Instructors should be able to send emails to selected groups of students
6. Support study replication
1. Support pre-existing analysis within the tool

9. GOAL
MOOCs
9
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
Develop a tool that will assist MOOC instructors/researchers in analyzing the MOOC data
Design decisions:
1. For now, desktop based
1. Later maybe provide a web-based version
2. Data sharing & IRB issues
2. No fancy database scheme abstractions, focus on one platform at a time
1. Coursera session-based implemented, edX in progress
3. Java-based to support multiple platforms

10. OVERVIEW TAB
MOOCito
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
10

11. OVERVIEW TAB
MOOCito
11
Lak17Kovanovic et al. – Developing a MOOC experimentation platform

12. DETAILS TAB
MOOCito
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
12

13. ANALYZE TAB
MOOCito
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
13

14. ANALYZE | INDICATORS TAB
MOOCito
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
14

15. ANALYZE | RUN ANALYSIS TAB
MOOCito
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
15

16. ANALYZE | EVALUATE TAB
MOOCito
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
16

17. ANALYZE | EVALUATE TAB
MOOCito
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
17

18. INTERVENE TAB
MOOCito
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
18

19. EVALUATION
MOOCito
Lak17Kovanovic et al. – Developing a MOOC experimentation platform
19
1. Conducted a user study with 11 participants (UK, USA, South America)
1. All participants were experienced with online learning, MOOCs, and Coursera platform
2. Case scenario of a hypothetical MOOC instructor halfway through the course
3. Survey questionnaire based on technology acceptance model (TAM)

20. STUDY
MOOCito
20
Lak17Kovanovic et al. – Developing a MOOC experimentation platform

21. RESULTS
MOOCito
1. Overall, participants were satisfied with the proposed system and its perceived usability and
ease-of-use.
2. Most positive responses:
1. The ability to generate interesting questions and hypotheses (4.73),
2. The ability of the tool to provide insights into students’ engagement with the learning
environment (4.64), and
3. The capacity to provide different interventions to subgroups of students (4.55).
3. In open-ended responses, participants emphasized:
1. The selection of engagement indicators, weekly plots, and clear interface.
2. The flexibility and ease of performing interventions without “the dangers of providing
blanket information inappropriately” to a particular target subset of students
3. The use of templates and “variables” in the message body.
4. Strong willingness (4.36) to use MOOCito and would like to be able to use it in their courses
(4.00).
1. Willingness to use it for research was higher (4.55) than for course design and teaching
purposes (4.30).
21
Lak17Kovanovic et al. – Developing a MOOC experimentation platform

22. ISSUES
MOOCito
1. Concerns regarding the accessibility of the tool for “typical” MOOC instructors.
1. Not familiar with: 1. Terminology, 2. Engagement indicators, 3. Statistical analyses, or 4.
Study “templates”
2. Future work: 1. Short summaries, 2. Video tutorials and 3. Bibliographical information
2. What would be the best way to analyze the data?
1. Future work: Additional information on the “standard approaches”
3. Simplification of the analysis steps (D4)
1. Future work: More streamlined analysis, better guidelines and support for instructors
with little background and experience in statistics.
4. Instructors should be able to go from indicators of engagement to the actual student
content.
5. Challenge of knowing when or how to intervene,
1. The need for more “interpretable” results, summarized in a form which is easy to act
upon.
2. Simple descriptions using “low,” “moderate,” and “high” for describing engagement
22
Lak17Kovanovic et al. – Developing a MOOC experimentation platform

23. PROGRESS
MOOCito
1. Support for edX and Coursera rolling session format
2. Work on user interface
3. Work on SNA analysis
4. Modularizing it so it is extensible
5. “Interpretable results”
23
Lak17Kovanovic et al. – Developing a MOOC experimentation platform

24. SUMMARY
MOOCito
“This has the potential to be a powerful tool in educational research as well
as a means of tracking learner engagement day-to-day.”.
24
Lak17Kovanovic et al. – Developing a MOOC experimentation platform

25. THANK YOU
QUESTIONS?
Web: http://vitomir.kovanovic.info
Mail: v.kovanovic@ed.ac.uk
Twitter: @vkovanovic
LUNCH IS GETTING COLD
The end
25
Lak17Kovanovic et al. – Developing a MOOC experimentation platform