RESET meeting 07/06/2018 Barcelona, Spain

1. Ishari Amarasinghe
Thesis Supervisors: Davinia Hernández-Leo & Anders Jonsson
TIDE, Universitat Pompeu Fabra, Barcelona
Sustaining Continuous Collaborative Learning
Flows in MOOCs: Orchestration Agent Approach
twitter: @TIDE_UPF
website : http://www.upf.edu/web/tide
1
RESET & Smarlet meeting June 07th, 2018

2. Introduction
2
● Collaborative Learning Opportunities in MOOCs [Manathunga et al. 2017]
● Sustaining learners' engagement in these spaces is challenging due to
different reasons [Yang et al. 2013]
● CSCL scripts aim to structure social interactions [Dillenbourg and Tchounikine 2007]
○ Collaborative Learning Flow Patterns (CLFPs) formulate the essence of
script structures [Hernández-Leo et al. 2010]
■ e.g., Pyramid CLFP
● Suitable real-time management or the orchestration [Dillenbourg and Tchounikine 2007]
of collaboration is vital
○ To maintain successful & uninterrupted collaboration flows
○ To maintain pedagogical method structure 2

3. Adaptive and Intelligent Techniques in MOOCs
● Adaptive techniques in MOOCs platforms
■ Adaptive indicators to implement adaptivity within MOOCs context [Leris et al.
2017]
■ Cloud computing architectures for adaptive MOOCs (aMOOC) [Sonwalkar 2013]
● Intelligent techniques in MOOCs platforms
○ Agent based approaches
■ Pedagogical agents [Bendou et al. 2017]
■ Conversational agents [Ferschke et al.2015, Wen 2015]
○ Future research potential
■ Only a few intelligent tools & techniques have been tested and deployed into
the actual MOOCs context [Bassi et al. 2014, Rosé and Ferschke 2016, Fauvel and Yu 2016]
3

4. An exploratory study of Pyramid collaborative
learning activities in a MOOC
● Objective:
○ How individual participation differences affect collaborative learning flows (structured based on
CLFPs) deployed in MOOC contexts
● Tool:
○ PyramidApp [Manathunga and Hernández-Leo 2017]
Figure 1: A screenshot of the PyramidApp showing rating space (left) and the negotiation space (right) 4

5. An exploratory study of Pyramid collaborative
learning activities in a MOOC - Experimental Design
● Experimental Design
○ “3D Graphics for Web Developers” (FutureLearn MOOC platform)
○ Pyramid Activities - 3 consecutive weeks
○ 99 participants
Table 1: Pyramid activity configurations
Week Min.
students
per
pyramid
No.of
rating
levels
Small
group
size
Option sub.
time limit
Rating
sub. time
limit
No.of
pyramids
1 8 2 2 18h 18h 4
3 4 2 2 18h 18h 2
5 4 2 2 18h 18h 1
Option submission phase
Rating and
discussion
5

6. ● Overall Activity Participation
○ No. of participants and their engagement decreased over time
An exploratory study of Pyramid collaborative
learning activities in a MOOC - Results & Analyses
Figure 2: MOOC Pyramid activity participation
6

7. ● Individual student-level analysis
○ Participation differences across different phases of Pyramids
○ Inspired by the work already done in the field [Milligan et al. 2013, Alario-Hoyos et al. 2014]
An exploratory study of Pyramid collaborative
learning activities in a MOOC - Results & Analyses
Category Description
Lurkers Only logged into PyramidApp, no participation in any levels of the Pyramid
Contributors Participated in all levels of the collaborative learning activity
Initiators Participated only in the initial option submission stage
Raters Participated only in rating levels
Runners Participated in initial level and at least one rating level
Table 2 : Participant categorization
7

8. ● Individual student-level analysis
● Participation in Chat
○ Students have used chat only during the first week
An exploratory study of Pyramid collaborative learning
activities in a MOOC - Results & Analyses Contd.
Figure 3:Individual student participation in different Pyramid activities
8

9. ● Based on overall engagement analysis:
○ A majority of learners falls into the category of “Lurkers”
○ Lack of “Contributors”, “Raters” and “Runners”
● Chat Participation:
○ Help seeking behavior
● Choice of collaborative script design parameters:
○ Lack of support towards dynamic script parameters
● Online survey results on overall collaborative learning experience:
○ Lack of participation of peers resulted in isolation
■ e.g., “..no one replied to my questions at all..”
● Other concerns:
○ Individual differences might have an influence on Pyramid activity participation
An exploratory study of Pyramid collaborative learning
activities in a MOOC - Summary
9

10. ● Maintaining learners continuous engagement in collaborative learning spaces in MOOCs
is challenging
● Damaged flows affect enthusiastic students
○ E.g., Contributors
● Collaborative learning activities deployed in MOOCs:
Requires careful orchestration of collaborative script design parameters
Continuous interaction & feedback generation mechanisms
An exploratory study of Pyramid collaborative learning
activities in a MOOC - Summary Contd.
10

11. Empirical study for the design of an Orchestration
Agent for Pyramid activities in MOOCs
● Orchestration Agent intervention in PyramidApp
● Wizard of Oz (WOZ) Study to clarify design requirements [Maulsby et al. 1993]
● Experimental Design
○ “Innovative collaborative learning with ICT” (Canvas Network platform)
○ Tool: PyramidApp [Manathunga and Hernández-Leo 2017]
○ Pyramid activities: 1st and 2nd week of the MOOC
○ 92 participants
11

12. Empirical study for the design of an Orchestration
Agent for Pyramid activities in MOOCs
● Experimental Design
12
Activity
Type
Min. students
per pyramid
No.of rating
levels
Small group
size
Option sub.
time limit
Rating sub.
time limit
No.of pyramids
Very Rapid 4 - 6 2 2 12 mints. 12 mints. 11
Rapid 4 - 6 2 2 47 mints. 47 mints. 5
Long 4 - 6 2 2 2 h. 2 h. 10
Very Long 4 - 6 2 2 6 h. 6 h. 2
Table 3: Pyramid activity configurations

13. Empirical study for the design of an Orchestration
Agent for Pyramid activities in MOOCs Contd.
Figure 4: Orchestration Agent interventions in PyramidApp
13

14. Orchestration Agent interventions in PyramidApp
Results & Analyses
Figure 5: Patterns of engagement in different types of Pyramid activities - chord diagrams 14

15. Orchestration Agent interventions in PyramidApp
Summary
● Orchestration agent intervention to fulfill the minimum
student count requirement to generate a Pyramid
○ Very Rapid activities - 63.63 %
○ Rapid activities - 20%
○ Long activities - 60%
○ Very Long activities - 100%
● Initial option submission stage
○ Very Rapid activities - 18.18 %
● First rating submission stage
○ Mandatory & optional intervention across all activity
types
● Second rating submission stage
○ Mandatory intervention across all activity types
15

16. Orchestration Agent interventions in PyramidApp
Summary
● Orchestration agent intervention to fulfill the minimum student count
requirement to generate a Pyramid
○ Very Rapid activities - 63.63 %
○ Rapid activities - 20%
○ Long activities - 60%
○ Very Long activities - 100%
● First rating submission stage
○ Mandatory agent intervention was required across all activity types
○ Optional agent intervention was also required across all activity types
to create meaningful collaborations
● Second rating submission stage
○ very rapid and very long activities required a higher intervention
Orchestration agent intervention in Pyramid activities is important to
maintain uninterrupted yet meaningful collaborative learning flows
16

17. Conclusions & Future Work
17
● What we know:
○ Social learning spaces in MOOCs are beneficial
○ CLFPs pre-structure collaboration
● Exploratory MOOC case study and WOZ study findings:
○ Sustaining continuous collaborative learning flows in MOOCs are hard
○ Different participation behaviors and rigid script design parameters can have a
major affect
○ Agent intervention - possible directions for further research
● What we research further:
○ Different intelligent techniques
○ Orchestration Agent Dashboard
○ User perception (Human-Agent Perspectives)
Amarasinghe, I., Hernández-Leo, D., Manathunaga. K., Jonsson, A. Sustaining Continuous Collaborative Learning Flows in MOOCs: Orchestration
Agent Approach. In Journal of Universal Computer Science - New trends in Massive Open Online Courses (2018) (accepted with minor revisions)

18. References
● Alario-Hoyos, C., Pérez-Sanagustín, M., Delgado-Kloos, C., Hugo, A., Parada, G., Muñoz-Organero, M.: ``Delving into participants' profiles and use of social tools in
MOOCs"; IEEE Transactions on Learning Technologies, 7, 3 (2014), 260-266.
● Bassi, R., Daradoumis, T., Xhafa, F., Caballé, S., Sula, A.: ``Software agents in large scale open e-learning: a critical component for the future of massive online courses
(MOOCs)"; Proc. INCoS (International Conference on Intelligent Networking and Collaborative Systems), (2014), 184-188.
● Bendou, K., Megder, E., Cherkaoui, C.: ``Animated Pedagogical Agents to Assist Learners and to keep them motivated on Online Learning Environments (LMS or
MOOC)"; International Journal of Computer Applications, 168, 6 (2017), 46-53.
● Dillenbourg, P., Tchounikine, P.: ``Flexibility in macro-scripts for computer-supported collaborative learning"; Journal of Computer Assisted Learning, 23, 1 (2007),
1-13.
● Dillenbourg, P., Tchounikine, P.: ``Flexibility in macro-scripts for computer-supported collaborative learning"; Journal of Computer Assisted Learning, 23, 1 (2007),
1-13.
● Fauvel, S., Yu, H.: ``A Survey on Artificial Intelligence and Data Mining for MOOCs" (2016), arXiv:1601.06862v1 [cs.AI].
● Ferschke, O., Yang, D., Tomar, G., Rose, C.P.: ``Positive Impact of Collaborative Chat Participation in an edX MOOC"; Proc. International Conference on Artificial
Intelligence in Education, (2015), 115-124
● Hernández-Leo, D., Asensio-Pérez, J. I., Dimitriadis, Y., Villasclaras Fernández, E. D.: ``Generating CSCL scripts: from a conceptual model of pattern languages to the
design of real scripts"; Technology-enhanced learning: design patterns and pattern languages, (2010), 49-64.
● Lers, D., Sein-Echaluce, M.L., Hernández, M., Bueno, C.: ``Validation of indicators for implementing an adaptive platform for MOOCs'': Computers in Human Behavior,
72 (2017), 783-795.
● Maulsby, D., Greenberg, S., Mander, R.: ``Prototyping an intelligent agent through Wizard of Oz"; Proc. INTERACT'93 and CHI'93 conference on Human factors in
computing systems, ACM, (1993), 277-284.
● Milligan, C., Littlejohn, A., Margaryan, A.: ``Patterns of Engagement in Connectivist MOOCs"; Journal of Online Learning and Teaching, 9, 2 (2013), 149-159.
● Manathunga, K., Hernández-Leo, D., Sharples, M.: ``A Social Learning Space Grid for MOOCs: Exploring a FutureLearn Case"; Proc. European Conference on Massive
Open Online Courses (EMOOCs), (2017), 243-253.
● Manathunga, K., Hernández-Leo, D: ``Authoring and Enactment of Mobile Pyramid-based Collaborative Learning Activities"; British Journal of Educational Technology,
49, 2 (2018), 262-275.
● Rose, C. P., Ferschke, O.: ``Technology Support for Discussion Based Learning: From Computer Supported Collaborative Learning to the Future of Massive Open Online
Courses"; International Journal of Artificial Intelligence in Education, 26, 2 (2016), 660-678.
● Sonwalkar, N.: ``The First Adaptive MOOC: A Case Study on Pedagogy Framework and Scalable Cloud Architecture--Part I.''; MOOCs Forum, 1, (2013), 22-29,
http://online.liebertpub.com/doi/pdfplus/10.1089/mooc.2013.0007
● Wen, M.: ``Investigating Virtual Teams in Massive Open Online Courses: Deliberation-based Virtual Team Formation, Discussion Mining and Support"; PhD diss.,
Stanford University (2015).
● Yang, D., Sinha, T., Adamson, D., Ros'e, C. P.: ``Turn on, Tune in, Drop out: Anticipating student dropouts in Massive Open Online Courses"; Proc. NIPS Data-driven 18

19. RESET & Smarlet meeting June 07th, 2018
Ishari Amarasinghe
Supervisors: Davinia Hernández-Leo & Anders Jonsson
TIDE, Universitat Pompeu Fabra, Barcelona
Sustaining Continuous Collaborative Learning
Flows in MOOCs: Orchestration Agent Approach
twitter: @TIDE_UPF
website : http://www.upf.edu/web/tide
19