Using games and mixed reality for understanding and communicating complex ideas: technical approaches and challenges
•
2 likes•210 views
Presentation used for talk at TU Graz - Graz University of Technology, on April 1st, 2019.
Recommended
Recommended
More Related Content
Similar to Using games and mixed reality for understanding and communicating complex ideas: technical approaches and challenges
Similar to Using games and mixed reality for understanding and communicating complex ideas: technical approaches and challenges (20)
More from Leonel Morgado
More from Leonel Morgado (20)
Recently uploaded
Recently uploaded (20)
Using games and mixed reality for understanding and communicating complex ideas: technical approaches and challenges
- 1. Using games and mixed reality for understanding and communicating complex ideas: technical approaches and challenges Leonel Morgado Leonel.Morgado@uab.pt April 1st, 2019
- 3. Let us go back to the distant past... …I’ll show ye things that ceased to be…
- 5. …but of these, only 2 remain in today’s games. 1985
- 6. Instruction manuals are gone! Games have to “teach their own players”
- 7. 1957
- 8. Bilbo Baggins as drawn by the author, J.R.R. Tolkien
- 10. How old is Bilbo, in these illustrations?
- 12. 20 30 40 50How do you look at the scene, thinking about different ages for Bilbo?
- 14. How many contradictory feelings and ideas are found in each perspective?
- 15. Picture from: "Urban subsystem" by Massimo.tadi - Own work. Licensed under CC BY-SA 4.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Urban_subsystem.jpg#/media/File:Urban_subsystem.jpg …are complex …systems …multi-layered …dynamic Cities and Society…
- 16. …and public participation is challenging WorkDaily life Pollution Infrastructure Traffic Costs
- 17. “…nothing is permanent except change; and the primary purpose of differential equations is to serve as a tool for the study of change in the physical world… these applications include: (…) The Einstein formula e=mc*c Newton’s law of gravitation The wave equation for the vibrating spring (…) The predator-prey equations (…) I consider the mathematical treatment of these problems to be among the chief glories of Western civilization…”
- 18. Can you explain to the public… …dynamics? …feedback response? Even to engineers and economists, outside their fields?
- 20. Cause and effect …seem simple repeat 4 [ forward 10 right 90 ]
- 21. learn square :side repeat 4 [ forward :side right 90 ] end repeat 36 [ square 100 right 10 ]
- 22. Emergence: it´s hard to… …foresee …understand …accept.
- 25. What will this second pass do?
- 27. Emergence: it´s hard to… …foresee …understand …accept.
- 34. Darfur is Dying, 2006
- 36. Cruz G. et al. (2013) Designing User Learning Experience in Virtual Worlds: The Young Europeans for Democracy Serious Application.
- 40. From end-users and children to professionals… anyone can participate in game design. You don’t even need computers!
- 41. Technical Approaches and Challenges
- 42. And again let’s go to the distant past…
- 43. …of 2007 Could we do this every week? A full semester?
- 45. …but we did not anticipate the bureaucracy. Keep track of class attendance!
- 46. I’m Andabata Mandelbrot, your professor. …Did anyone just arrive? …Did anyone just leave?
- 48. Besieged by chores! Has the student progressed? What did each student deliver? How do I hand them objects? Where can I find each student’s work?
- 49. - How do we ensure that only the enrolled students participate? - How can a training manager check what trainers are doing, i.e., how can we manage training sessions? - Who sets up a virtual space? Can it be automated? http://hdl.handle.net/10400.2/5454
- 50. Don’t just create – integrate! For widespread use… …of games …virtual worlds …mixed reality…
- 51. Meanwhile… someone asked about teams.
- 52. Define once: replay anytime Use of a virtual world system in sports coach education for reproducing team handball movements https://journals.tdl.org/jvwr/index.php/jvwr/article/view/399/457
- 53. How do we avoid lock-in with a specific platform? Knowledge on tactics will outlast the technological platform!
- 55. A new type of data: By matching xAPI recipes… …choreographies can move across platforms http://repositorioaberto.uab.pt/handle/10400.2/3487 Model-driven generation of multi-user and multi-domain choreographies for staging in multiple virtual world platforms
- 56. Can we reach widespread adoption? Exploring educational immersive videogames: an empirical study with a 3D multimodal interaction prototype http://repositorioaberto.uab.pt/handle/10400.2/5815
- 57. 1 Access 3 Deployment 2 Production For widespread adoption, we need to break these challenges Technology Challenges of Virtual Worlds in Education and Training - Research Directions http://repositorio.utad.pt//handle/10348/2787
- 58. 1 Access 3 Deployment 2 Production For widespread adoption, we need to break these challenges Technology Challenges of Virtual Worlds in Education and Training - Research Directions http://repositorio.utad.pt//handle/10348/2787 Make it FEASIBLE for any educator or student to use serious games and virtual worlds anyday, anytime, often, and yearlong. Not just occasionally, as a gimmick or only in the hands of enthusiasts!
- 62. Can we do it? …but we need to try harder! https://immersivelrn.org/ilrn2019/ Let’s discuss! Leonel.Morgado@uab.pt
Editor's Notes
- Let me tell you a story, far in the distant past…
- …2007, 10 years ago. I was using Second Life to teach computer programming, along with one of my PhD students, Micaela Esteves. We were getting nice results, using it as a virtual toy programming platform, simulating shops for customer-tracking backoffice code, etc. So, naturally, we wanted to try it in more ambitious settings. What if a full class used virtual worlds for a semester, twice a week, every week?
- The problem was that my PhD student lived xxxx km away, where she was a teaching assistant, and couldn’t be physically present twice a week. That’s OK, I figured, why don’t you conduct the classes entirely online in Second Life? Why travel at all? Where’s a spoiler: it worked, we learned a lot about how to do it, what happened, and you can actually find more details in a BJET paper published some years ago.
- But there was a problem implementing it that we did not anticipate… …bureaucracy. “You need to keep track of attendance”, they said.
- So… I was Andabata Mandelbrot. Who was everyone else? Avatar names wouldn’t match student names… But virtual environments are more fluid. When someone enters this room, or leaves it, we notice it. As in a physical classroom. But in a virtual world… people may “pop in” or “pop up” for lots of different reasons. Perhaps their network failed. Perhaps they weren’t interested and left. Perhaps they arrived late… but there is not virtual door opening. There is no gush of air, no sound of footsteps leaving, someone simply appears or disappears… When did the student arrive? When did the student leave? What was missed? Was it a moment, or a large piece of the lesson?
- We did it by hand, as best as we could. Meanwhile, I supervised a Master student that came up with was a Moodle module to keep track of a minute-by-minute attendance of a class, and associate each student with its virtual world avatar. This generated an attendance chart for each student, showing when was the student present in the class, and allowing the teacher to decide if it could count as attending or not. It also allowed the teacher to realize what a student might have missed.
- At the same time, we were feeling the chores of managing all the tasks in virtual world classes: Has the student progressed? What was delivered by each student? How do I deliver an object or component to each student? How do I track to whom I’ve already delivered components? Where, exactly, is a student working, within the virtual space? How do I provide contextual feedback? Do I have to give instructions on how to reach the location of an issue, instead of commenting there? Why can’t I annotate in 3D as in a shared Google Doc?
- Altice Labs (at the time, called Portugal Telecom Innovation), asked us to help them use 3D virtual worlds in professional training. They were having much the same concerns, including aspects such as: How do we ensure that only the enrolled students participate? How can a training manager check what trainers are doing, i.e., how can we manage training sessions? Can we reuse and assign 3D objects to various training sessions, as we do with documents, presentations, videos or images? Who sets up a virtual space? Can it be automated? Can the notification/logging in/logging out workflow be automated, too?
- This made us realize the importance of integration of virtual worlds and games with learning management systems. If games and virtual worlds – and indeed AR and Mixed Reality – are not embedded in the daily tasks of students, teachers, and managers, it’s hard to envision how they can become widespread. I believe they will, eventually, but their adoption rate, their dissemination will be slower – much slower than we desire.
- Another happy circumstance occurred, at around the same time. I was approached by a Sports Sciences PhD student (who I ended up co-supervising) who had a radical new idea for training handball coaches on team tactics: use a virtual world. This way, you could teach the “perfect”, conceptualized idea of a tactic. No more depending on videotaping that lacked the proper perspective, or on endless rehearsing a tactic so that no player is early or late on the spot, looking the wrong way, gesturing unnecessarily, etc. But how could that Sports Sciences student define the tactics without the help of programmers?
- We came up with a “magnet board” application which allowed the Sports Sciences student to drag markers on a timeline, and thus define the tactics. Then the resulting file was read by a replay program that would control 14 Second Life bots and replay the tactic upon request, with 3D avatars. The coaches during training sessions would collectively watch the tactic, in 3D, change the perspective at will, and the trainer could pause it, resume it, etc. They could even position themselves amidst the team, in order to better argue on a problem or new idea.
- We were, however, concerned about the lock-in of our system with Second Life or OpenSimulator. The tactics were defined outside these systems, but the replay software was tied to its programming interfaces, the very definition of motion and events depended on Second Life / OpenSimulator’s concepts of motion, such as “activating a gesture”, moving “forward” for a few meters, etc. Knowledge on handball tactics will probably last more than the few years a given technological platform lasts! What about for research? If we encode tactics as we study teams in the physical world, e.g., the tactics of teams in a World Championship, do we tie them to the specific replay requirements of a game platform that will inevitably become outdated?
- Shouldn’t we be able to pick a tactic from 2010, then replayed in Second Life, and be able to witness it in mixed reality today, displayed before our eyes, overlaid on a physical handball court? Sure, we’d like to use better-rendered players, but that’s the least of our concerns. Even if we had to make do with default avatars, even if we had to make do with 3D Monopoly game pieces, wouldn’t it be better than to simply lose our handball tactics data, lose our 3D educational content, lose our research records, only because technology evolved?
- We’ve started working on this problem, and realized that games and virtual worlds – and indeed AR and mixed reality – should witness the emergence of a new type of data: virtual choreographies. That is, we should be able to let teachers and students define what happens in a game platform – which we already do, by playing the game – but reuse those events, those behaviors, for analysis, for discussion and review of ideas, for providing new students with our collection of prepared learning situations. And not just loose all of it when we move from a old technological platform for a new one. We can imagine a library of virtual choreographies – an “app store” of educational situations – that are shared among students, teachers, and across different games, AR & Mixed Reality platforms, that survive the obsolescence of their original rendering technologies. Knowledge must survive its technological platforms. Knowledge must be as eternal as its truthfulness. Virtual choreographies are our proposed solution for this.