The document discusses the use of augmented reality (AR) in creating empathic experiences and enhancing communication by removing technological barriers and changing perspectives. It highlights concepts such as emotional intelligence, empathic computing, and various applications of AR, including remote collaboration and social panoramas. The conclusion emphasizes that AR can facilitate shared experiences, which are essential for emotional connections, and identifies future research directions in scaling these applications.

1. Using Augmented Reality to Create
Empathic Experiences
Mark Billinghurst
mark.billinghurst@hitlabnz.org
The HIT Lab NZ, University of Canterbury
February 27th 2014

5. Courtesy Matt Rettig, CMU

18. Processing Power
Adapt
Experience
Operate

19. Beyond the Desktop

20. Intelligent User Interfaces
 AI + HCI: User Interface involving some
elements of Artificial Intelligence
 Computer having model of user/domain
 First IUI Conference in 1997
 Readings in IUI (Wahster 1998)

21.  Microsoft Clippy (1997)
 MS Office Intelligent User Interface

23. Intelligent User Interfaces

24. Multiple Intelligences
 Frames of Mind: The
Theory of Multiple
Intelligences
 Howard Gardner (1983)

25. Multiple Intelligences

26. Emotional Intelligence
 Emotional Intelligence - Why it
can matter more than IQ
 Goldman (1995)
 Identify, assess, and control
the emotions of oneself, of
others, and of groups

27. Foundations of Emotional Intelligence

28. Empathy
Empathy

30. Empathy vs. Intelligence
 Intelligence: the power of one’s brain, divided
into many categories and used in numerous ways.
 Empathy: the power of one’s heart, expressing
one’s true emotions with themselves, those
around them, and their own world.

31. IQ vs EQ?

32. Mirror Neurons
 Neuron that fires both when an animal acts and it
observes the same action performed by another
 Giacomo Rizzolatti, Univ. of Palma (1980s/90s)

34. Empathic Computing
1. Computing systems that can
understand your feelings and emotions
2. Computing systems that help you
better understand the feelings of others

35. Affective Computing
 Ros Picard – MIT Media Lab
 http://affect.media.mit.edu

36. Appliances That Make You Happy
 Jun Rekimoto – Univ. Tokyo
 Smile detection + smart devices

37. Can we develop interfaces
that allow us to be more
empathetic to others?

38.  adsf

39. Empathy Computing Requirements
 Basic Requirements
 Making the technology transparent
 Empathy Definition
 Seeing with the eyes of another
 Hearing with the ears of another
 Feeling with the heart of another

40. Augmented Reality
1977
2008

41. Using AR for Empathy
 Augmented Reality can:
 Remove technology barriers
 Enhance communication
 Change perspective
 Share experiences
 Enhance interaction in real world

42. Communication Seams
Communication Space
Task Space
 Technology introduces artificial seams in the
communication (eg separate real and virtual space)

43. Removing Barriers: Shared Space
 Face to Face interaction, Tangible AR metaphor
- ~3,000 users (Siggraph 1999)
 Easy collaboration with strangers
 Users acted same as if handling real objects
Billinghurst, M., Poupyrev, I., Kato, H., & May, R. (2000). Mixing realities in shared space: An augmented
reality interface for collaborative computing. In Multimedia and Expo, 2000. ICME 2000. 2000 IEEE
International Conference on (Vol. 3, pp. 1641-1644).

44. Enhancing Face to Face Communication
 AR Pad
 Handheld AR device
 AR shows viewpoints
 Users collaborate easier
 Show communication cues
Virtual Viewpoint Visualization
Mogilev, D., Kiyokawa, K., Billinghurst, M., & Pair, J. (2002, April). AR Pad: An interface for face-to-face AR
collaboration. In CHI'02 extended abstracts on Human factors in computing systems (pp. 654-655).

45. Changing Perspective
 CamNet (1992)
 British Telecom
 Wearable Teleconferencing
 audio, video
 Remote collaboration
 Sends task space video
 Similar CMU study (1996)
 cut performance time in half

46. WACL: Remote Expert Collaboration
 Wearable Camera/Laser Pointer
 Independent pointer control
 Remote panorama view

47. WACL: Remote Expert Collaboration
 Remote Expert View
 Panorama viewing, annotation, image capture
Kurata, T., Sakata, N., Kourogi, M., Kuzuoka, H., & Billinghurst, M. (2004, October). Remote collaboration
using a shoulder-worn active camera/laser. In Wearable Computers, 2004. ISWC 2004. Eighth
International Symposium on (Vol. 1, pp. 62-69).

49. View Through Google Glass
Always available peripheral information display
Combining computing, communications and content capture

50. Ego-Vision Collaboration
 Google Glass
 camera + processing + display + connectivity

51. Current Collaboration on Glass
 First person remote conferencing/hangouts
 Limitations
 Single POV, no spatial cues, no annotations, etc

52. Sharing Space: Social Panoramas
 Capture and share social spaces in real time
 Enable remote people to feel like they’re with you

53. Key Technology
 Google Glass
 Capture live image panorama (compass + camera)
 Capture spatial audio, live video
 Remote device (desktop, tablet)
 Immersive viewing, live annotation

54. Capturing Space: Real World Capture
 Hands free AR
 Portable scene capture (color + depth)
 Projector/Kinect combo, Remote controlled pan/tilt
 Remote expert annotation interface

55. Remote Expert View

56. Capturing Behaviours
 3 Gear Systems
 Kinect/Primesense Sensor
 Two hand tracking
 http://www.threegear.com

57. Skeleton Interaction + AR
 HMD AR View
 Viewpoint tracking
 Two hand input
 Skeleton interaction, occlusion

58. Ghostman
 Use AR to capture and overlay your actions
into remote persons space
 Eg remote therapy

59. Looking to the Future
What’s Next?

60. Brain to Brain Control
 Rajesh Rao, University of Washington
 First Brain to Brain control

61. System Architecture

62. Scaling Up
 Seeing actions of millions of users in the world
 Augmentation on city/country level

63. AR + Smart Sensors + Social Networks
 Track population at city scale (mobile networks)
 Match population data to external sensor data
 medical, environmental, etc
 Mine data to improve social services

66. Research Challenges
 How to convey emotion?
 How to measure empathy?
 Interface/interaction models?
 How to communicate emotion?
 Scaling up to city/country scale?

67. Conclusion

68. Harvard Grant Study
 $20 million, 75 years study
 268 Harvard graduates
 456 disadvantaged people
 Led by George Valliant
 What makes us happy?
 warmth of relationships throughout
life have the greatest positive impact
on "life satisfaction".

69. “The seventy-five years and twenty million dollars
expended on the Grant Study points to a
straightforward five-word conclusion: Happiness is
love. Full stop.”
George Valliant

70. Conclusions
 Empathic Computing
 Sharing what you see, hear and feel
 AR Enables Empathic Experiences
 Removing technology
 Changing perspective
 Sharing space/experience
 Many directions for future research

71. More Information
• Mark Billinghurst
– Email: mark.billinghurst@hitlabnz.org
– Twitter: @marknb00
• Website
– http://www.hitlabnz.org/