Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Multimodal Machines #JTELSS17 workshop


Published on

Here the slides of the workshop "Multimodal Machines" presented at the JTEL Summer School 2017 in Aveiro, Portugal. For more info about it

Published in: Education
  • Be the first to comment

  • Be the first to like this

Multimodal Machines #JTELSS17 workshop

  1. 1. Workshop @ JTELSS17 12th October 2017 – Aveiro, Portugal Multimodal Machines Daniele DI MITRI & Jan SCHNEIDER
  2. 2. What is multimodality? Pagina 2 Group assignment: divide into groups and find a picture which best describe the concepts of Multimodality.
  3. 3. What is multimodality? Pagina 3
  4. 4. Pagina 4 How does multimodality work in learning?
  5. 5. Multimodality for robots Pagina 5 Group assignment: equip your robot with multimodal capabilities
  6. 6. They encode messages Pagina 6 Multimodality for computers e.g. 3D models in AR They decode sensor inputs e.g modern AR headsets like Hololenes
  7. 7. Observability Line INPUT SPACE OUTPUT SPACE Observable dimensions, can be tracked with sensors Unobservable dimensions, require human interpretation assessment Di Mitri, D., Drachsler, H., Specht, M. (2017) From signals to knowledge. A conceptual model for multimodal learning analytics. In press.
  8. 8. Multimodal data tree Pagina 8
  9. 9. Famous multimodal experiments
  10. 10. Put-that-there Pagina 10 Bolt, R. A. (1980). “Put-that-there”: Voice and gesture at the graphics interface (Vol. 14, No. 3, pp. 262-270). ACM.ISO 690
  11. 11. Emotion sensors go to school Pagina 11 Arroyo, I., Cooper, D. G., Burleson, W., Woolf, B. P., Muldner, K., & Christopherson, R. (2009, July). Emotion Sensors Go To School. In AIED (Vol. 200, pp. 17-24)
  12. 12. Multimodal action based assessment Pagina 12 Worsley, M., & Blikstein, P. (2013, April). Towards the development of multimodal action based assessment. In Proceedings of the third international conference on learning analytics and knowledge (pp. 94-101). ACM.ISO 690
  13. 13. Analytics meet Patient Manikins Pagina 13 Martinez-Maldonado, R., Power, T., Hayes, C., Abdipranoto, A., Vo, T., Axisa, C., and Buckingham-Shum, S. (2017) Analytics Meet Patient Manikins: Challenges in an Authentic Small-Group Healthcare Simulation Classroom. International Conference on Learning Analytics and Knowledge, LAK 2017, 90-94.
  14. 14. Multimodal experiments @OUNL
  15. 15. Learning Pulse – are you in the Flow? Pagina 15 (Csikszentmihalyi, 1972) Di Mitri, D., Scheffel, M., Drachsler, H., Börner, D., Ternier, S., & Specht, M. (2017). Learning Pulse: a machine learning approach for predicting performance in self-regulated learning using multimodal data.ISO 690
  16. 16. Presentation Trainer Pagina 16 Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015, November). Presentation trainer, your public speaking multimodal coach. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (pp. 539-546). acm.ISO 690
  17. 17. project Pagina 17
  18. 18. Calligraphy Learning Pagina 18
  19. 19. Q&A Thanks for listening! Daniele Di Mitri & Jan Schneider