Co-located Collaboration Analytics

1. Co-located Collaboration Analytics Sambit Praharaj with: Maren Scheffel, Hendrik Drachsler and Marcus Specht ICMI 2019 Doctoral Consortium Suzhou, China 14 Oct 2019

2. What is collaboration? • Occurs when two or more persons work towards a common goal (Martinez- Moyano, 2006) • Can be in different settings: – Online (or remote) – Co-located (or face-to-face) • Can be for different purposes: – Programming (Grover et al., 2016) – Meetings (Terken & Strum, 2010) – Problem solving (Cukurova et al., 2018) – Brainstorming (Tausch et al., 2014), etc. Pagina 2

3. MMLA in CC Pagina 3 Co-located Collaboration

4. Multimodal Indicators of CC • Non-verbal signals using NISPI framework (Cukurova et al., 2018) • Joint Visual Attention (JVA) (Richardson & Dale, 2005; Jermann et al., 2011; Schneider et al., 2015) • Hand movements, head movement and physical engagement (Spikol et al., 2017) • Audio cues (Bassiou et al., 2016) Pagina 4

5. Dimensions or Indexes of CC • Mutual regulation (Blaye, 1988), Conflict resolution (Doise et al., 1976) • IVA, IA, Synchrony, Equality (Spikol et al., 2017; Cukurova et al., 2017, 2018) • 9 dimensions (Meier et al. , 2007) and other dimensions (Johnson & Johnson, 2009) Pagina 5

6. Indicators  Dimensions or Indexes Onderwerp via >Beeld >Koptekst en voettekst Pagina 6

7. Real-time or post-hoc feedback • Reflect table (Bachour et al., 2010) • Conversation Clocks (Bergstrom & Karahalios, 2007a, 2007b) • Groupgarden (Tausch et al., 2014) • JVA (Schneider et al., 2015) Pagina 7

8. Real-time feedback Pagina 8

9. Current problems • Huge gap between the theory and practicality of the dimensions of collaboration • Most of the feedback systems provide a post-hoc feedback or a real- time reflective feedback • Dearth of studies on automated multimodal analysis in non-computer supported environments (Worsley & Blikstein, 2015) Pagina 9

10. Research Questions • RQ1 – What multimodal indicators give evidences of the quality of CC? • RQ2 – To what extent, can we design an automated real-time feedback system using the insights from the multimodal indicators to facilitate CC? Pagina 10

11. Research Challenges Pagina 11 Designing the collaboration task and it’s outcomes or decide the collaboration scenario Architectural design to integrate multiple sensors and enable real-time prediction Annotating the huge dataset collected

12. Research Tasks Pagina 12 • T1 – Literature Review: MMLA in CC • T2 – Formative Study: Prototype design, pilot study, collection of data • T3 – Summative Study: Evaluation and modification of the feedback strategy

13. Rough conceptual framework Pagina 13

14. Expected Outcomes Pagina 14 Theoretical indexes of CC Practically detected indexes of CC Post-hoc Real-time (Soller et al., 2005)

15. 1st Use-Case study Pagina 15 (Praharaj et al., 2018)

16. 2nd study Onderwerp via >Beeld >Koptekst en voettekst Pagina 16 (Praharaj et al., 2019)

17. 2nd study Onderwerp via >Beeld >Koptekst en voettekst Pagina 17

18. Thank you! “The greatest danger for most of us is not that our aim is too high and we miss it, but that it is too low and we reach it.” – Michelangelo Pagina 18 SCAN ME

19. Our team Pagina 19

20. References Bachour, K., Kaplan, F., & Dillenbourg, P. (2010). An interactive table for supporting participation balance in face-to-face collaborative learning.IEEE Transactions on Learning Technologies,3(3), 203–213. Baker, M. J. (1999). Argumentation and constructive interaction. Foundations of argumentative text processing, 5, 179–202. Bassiou, N., Tsiartas, A., Smith, J., Bratt, H., Richey, C., Shriberg, E., ... & Alozie, N. (2016, September). Privacy-Preserving Speech Analytics for Automatic Assessment of Student Collaboration. In INTERSPEECH (pp. 888-892). Bergstrom, T., & Karahalios, K. (2007a). Conversation clock: Visualizing audio patterns in co-located groups. In System sciences, 2007. hicss 2007. 40th annual hawaii international conference on(pp. 78–78). Bergstrom, T., & Karahalios, K. (2007b). Seeing more: visualizing audio cues. Human-Computer Interaction–INTERACT 2007, 29–42. Blaye, A. (1988). Confrontation socio-cognitive et r ́esolution de probl`eme (`a propos du produit de deux ensembles)(Unpublished doctoral dissertation). Blikstein, P. (2013). Multimodal learning analytics. Proceedings of the Third International Conference on Learning Analytics and Knowledge - LAK ’13 , 102. http://doi.org/10.1145/2460296.2460316 Chikersal, P. (2017). Deep structures of collaboration(Unpublished doctoral dissertation). Carnegie Mellon University Pittsburgh, PA. Pagina 20

21. References Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups.Review of educational research,64(1), 1–35. Craig, S. D., D’Mello, S., Witherspoon, A., & Graesser, A. (2008). Emote aloud during learning with autotutor:Applying the facial action coding system to cognitive–affective states during learning. Cognition and Emotion,22(5), 777–788. Cukurova, M., Luckin, R., Mavrikis, M., & Mill ́an, E. (2017). Machine and human observable differences in groups’ collaborative problem-solving behaviours. In European conference on technology enhanced learning(pp. 17–29). Cukurova, M., Luckin, R., Mill ́an, E., & Mavrikis, M. (2018). The nispi framework: Analysing collaborative problem-solving from students’ physical interactions. Computers & Education,116, 93–109. Davidsen, J., & Ryberg, T. (2017). “This is the size of one meter”: Children’s bodily-material collaboration. International Journal of Computer-Supported Collaborative Learning, 12(1), 65-90. Dede, C.: Comparing frameworks for 21st century skills. 21st century skills: Rethinking how students learn 20, 51-76 (2010) Pagina 21

22. References Dillenbourg, P.: What do you mean by collaborative learning? (1999) D’mello, S. K., Craig, S. D., Witherspoon, A., Mcdaniel, B., & Graesser, A. (2008). Automatic detection of learner’s affect from conversational cues.User modeling and user-adapted interaction,18(1), 45–80. Doise, W., Mugny, G., & Perret-Clermont, A.-N. (1976). Social interaction and cognitive development: Further evidence.European Journal of Social Psychology,6(2), 245–247. Grover, S., Bienkowski, M., Tamrakar, A., Siddiquie, B., Salter, D., & Divakaran, A. (2016). Multimodal analytics to study collaborative problem solving in pair programming. In Proceedings of the sixth international conference on learning analytics & knowledge(pp. 516–517). Jermann, P., Mullins, D., N ̈ussli, M.-A., & Dillenbourg, P. (2011). Collaborative gaze footprints: correlates of interaction quality. In Connecting computer-supported collaborative learning to policy and practice: Cscl 2011 conference proceedings.(Vol. 1, pp. 184–191). Johnson, D. W., & Johnson, R. T. (2009). An educational psychology success story: Social interdependence theory and cooperative learning. Educational researcher, 38(5), 365-379. Kulyk, O., Wang, J., & Terken, J. (2005, July). Real-time feedback on nonverbal behaviour to enhance social dynamics in small group meetings. In International Workshop on Machine Learning for Multimodal Interaction (pp. 150-161). Springer, Berlin, Heidelberg. Lubold, N., & Pon-Barry, H. (2014, November). Acoustic-prosodic entrainment and rapport in collaborative learning dialogues. In Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge (pp. 5-12). ACM. Pagina 22

23. References Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme for assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Collaborative Learning, 2(1), 63-86. Praharaj S., Scheffel M., Drachsler H., Specht M. (2018) Multimodal Analytics for Real-Time Feedback in Co-located Collaboration. In: Pammer-Schindler V., Pérez-Sanagustín M., Drachsler H., Elferink R., Scheffel M. (eds) Lifelong Technology-Enhanced EC-TEL 2018. Lecture Notes in Computer Science, vol 11082. Springer, Cham Richardson, D. C., & Dale, R. (2005). Looking to understand: The coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension. Cognitive science,29(6), 1045–1060. Schneider, B., & Blikstein, P. (2015). Unraveling students’ interaction around a tangible interface using multimodal learning analytics.Journal of Educational Data Mining,7(3), 89–116. Schneider, B., & Pea, R. (2013). Real-time mutual gaze perception enhances collaborative learning and collaboration quality.International Journal of Computer-supported collaborative learning,8(4), 375–397. Schneider, B., & Pea, R. (2014). Toward collaboration sensing. International Journal of Computer-Supported Collaborative Learning,9(4), 371–395. Schneider, B., Abu-El-Haija, S., Reesman, J., & Pea, R. (2013). Toward collaboration sensing: applying network analysis techniques to collaborative eye-tracking data. In Proceedings of the third international conference on learning analytics and knowledge(pp. 107–111) Soller, A., Martínez, A., Jermann, P., & Muehlenbrock, M. (2005). From mirroring to guiding: A review of state of the art technology supporting collaborative learning. International Journal of Artificial Intelligence in Education, 15(4), 261-290. Pagina 23

24. References Schneider, B., Sharma, K., Cuendet, S., Zufferey, G., Dillenbourg, P., & Pea, R. D. (2015). 3d tangibles facilitate joint visual attention in dyads. InProceedings of 11th international conference of computer supported collaborative learning(Vol. 1, pp. 156–165). Spikol, D., Ruffaldi, E., & Cukurova, M. (2017). Using multimodal learning analytics to identify aspects of collaboration in project-based learning. Philadelphia, PA: International Society of the Learning Sciences. Tausch, S., Hausen, D., Kosan, I., Raltchev, A., & Hussmann, H. (2014, October). Groupgarden: supporting brainstorming through a metaphorical group mirror on table or wall. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (pp. 541-550). ACM. Worsley, M., & Blikstein, P. (2015). Using learning analytics to study cognitive disequilibrium in a complex learning environment. In Proceedings of the fifth international conference on learning analytics and knowledge(pp. 426–427). Worsley, M., & Blikstein, P. (2015, March). Leveraging multimodal learning analytics to differentiate student learning strategies. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 360-367). ACM. Praharaj, S., Scheffel, M., Drachsler, H., & Specht, M. (2019, September). Group Coach for Co-located Collaboration. In European Conference on Technology Enhanced Learning (pp. 732-736). Springer, Cham. Pagina 24