Keynote talk given by Mark Billinghurst at the International Conference on Teaching and Learning in Education, March 3rd 2016, in Kuala Lumpur, Malaysia. Talks about the use of AR and VR to provide educational experiences.

1. CREATING IMMERSIVE AND
EMPATHIC LEARNING EXPERIENCES
Mark Billinghurst
mark.billinghurst@unisa.edu.au
March 2nd 2016
International Conference on Teaching and
Learning in Education (ICTLE2016)

3. Students
• zxcv

4. • Xcv z

5. New Tools – Old Methods
• One to many, broadcast model
• Same content repackaged

6. Making Technology Invisible
• Computers becoming smaller and ubiquitous
• Separation between the Digital and Physical worlds

7. Making Interfaces Invisible
Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented
interaction with real world environments. In Proceedings of the 8th Annual ACM Symposium on
User interface and Software Technology. UIST '95. ACM, New York, NY, 29-36.

8. Graphical User Interfaces
• Separation between real and digital worlds
• WIMP (Windows, Icons, Menus, Pointer) metaphor

9. Ubiquitous Computing
• Computing and sensing embedded in real world
• Particle devices, RFID, motes, arduino, etc

10. littleBits - http://littlebits.cc/
• Plug and play hardware components
• Sensors, input, output
• Rapid design with hardware

11. VIRTUAL REALITY

12. Virtual Reality
• ImmersiveVR
• Head mounted display, gloves
• Separation from the real world

13. Occulus Rift
• $350 USD
• 360 degree head tracking
• 100 degree field of view

14. • zXc

15. Google Expeditions
• Teacher lead VR experiences
• https://www.google.com/edu/expeditions/

16. • zxcv

18. • zxv

19. • zxcvz

20. AUGMENTED REALITY

21. 1977 – StarWars

22. Augmented Reality Definition
• Defining Characteristics [Azuma 97]
• Combines Real andVirtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

23. 2008 - CNN

24. • Put AR pictures here
Augmented Reality Examples

25. AR vsVR

26. CityViewARApplication
• Visualize Christchurch before the earthquakes

27. Digital Binocular Station
http://www.DigitalBinocularStation.com/

28. Case Study 3:AR Chemistry (Fjeld 2002)
• Tangible AR chemistry education
Fjeld, M., Fredriksson, J., Ejdestig, M., Duca, F., Bötschi, K., Voegtli, B., & Juchli, P. (2007, April). Tangible
user interface for chemistry education: comparative evaluation and re-design. In Proceedings of the
SIGCHI conference on Human factors in computing systems (pp. 805-808). ACM.

29. Goal: An AR application to test molecular
structure in chemistry
• Physical Components
• Real book, rotation cube, scoop, tracking markers
• Display Elements
• AR atoms and molecules
• Interaction Metaphor
• Build your own molecule

30. AR Chemistry Input Devices

32. EMPATHIC COMPUTING

33. Empathy
“Seeing with the Eyes of another,
Listening with the Ears of another,
and Feeling with the Heart of another..”
Alfred Adler

34. Empathic Computing
1. Understanding: Systems that can
understand your feelings and emotions
2. Sharing: Systems that help you better
understand the feelings of others

35. Understanding: Affective Computing
• Ros Picard – MIT Media Lab
• Systems that recognize emotion

36. Appliances That Make You Happy
• Jun Rekimoto – University of Tokyo/Sony CSL
• Smile detection + smart appliances

37. Happiness Counter Demo

38. Sharing
Can we develop systems
that allow us to share what
we are seeing, hearing and
feeling with others?

39. • sdfs

40. • sdfgs

42. • axcvxca

43. Movies are like a
machine that
generates Empathy
Roger Ebert

44. Using VR for Empathy
• USC Project Syria (2014)
• VR Experience of Terrorist attack in Aleppo

45. Using AR for Empathy
• Augmented Reality can:
• Remove technology barriers
• Enhance communication
• Change perspective
• Share experiences
• Enhance interaction in real world

46. Example: Google Glass
• Camera + Processing + Display + Connectivity
• Ego-Vision Collaboration (But with FixedView)

47. Current Collaboration onWearables
• First person remote conferencing/hangouts
• Limitations
• Single POV, no spatial cues, no annotations, etc

48. Social Panoramas (ISMAR 2014)
• Capture and share social spaces in real time
• Supports independent views into Panorama

49. Implementation
• Google Glass
• Capture live image panorama (compass + camera)
• Remote device (tablet)
• Immersive viewing, live annotation

50. User Interfaces
Glass View
Tablet View

51. Lessons Learned
• Good
• Communication easy and natural
• Users enjoy have view independence
• Very natural capturing panorama on Glass
• Sharing panorama enhances the shared experience
• Bad
• Difficult to support equal input
• Need to provide awareness cues

52. JackIn – Live Immersive Video Streaming
• Jun Rekimoto – University of Tokyo/Sony CSL

53. JackIn Hardware
• Wide angle cameras – 360 degree video capture
• Live video stitching

54. JospehTame – Tokyo Marathon
• Live streaming from Tokyo marathon
• http://josephta.me/en/tokyo-marathon/

55. • zxcz

56. Empathic Glasses
• Combine together eye-tracking, display, face expression
• Impicit cues – eye gaze, face expression
++
Pupil Labs Epson BT-200 AffectiveWear

57. AffectiveWear – Emotion Glasses
• Photo sensors to recognize expression
• User calibration
• Machine learning
• Recognizing 8 face expressions
Masai, K., Sugiura, Y., Suzuki, K., Shimamura, S., Kunze, K., Ogata, M., ... & Sugimoto, M. (2015,
September). AffectiveWear: towards recognizing affect in real life. In Proceedings of the 2015 ACM
International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM
International Symposium on Wearable Computers (pp. 357-360). ACM.

58. Integrated System
• Local User
• Video camera
• Eye-tracking
• Face expression
• Remote Helper
• Remote pointing

59. System Diagram
• Two monitors on Remote User side
• Scene + Emotion Display

60. Lessons Learned
• Pointing really helps in remote collaboration
• Makes remote user feel more connected
• Gaze looks promising
• shows context of what person talking about
• More work needed on emotion/expression cues
• Limitations
• Limited implicit cues
• Two separate displays
• Task was a poor emotional trigger
• AffectiveWear needs improvement

61. FUTURE RESEARCH

62. Looking to the Future

63. Scaling Up
• Seeing actions of millions of users in the world
• Augmentation on city/country level

64. AR + Smart Sensors + Social Networks
• Track population at city scale (mobile networks)
• Match population data to external sensor data
• Mine data for applications

65. Example: MIT SENSEable City Lab
http://senseable.mit.edu/wikicity/rome/

66. Example: CSIRO WeFeel Tool
• Emotionally mining global Twitter feeds
• http://wefeel.csiro.au

67. CONCLUSION

68. Summary
• There is a need for new teaching tools
• Making technology invisible when presenting content
• VR and AR provide innovative opportunities
• Virtual Reality
• Immersive experience
• Augmented Reality
• Enhanced interaction in real world
• Empathic Computing
• Sharing experience, understanding

69. www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au