The document discusses research directions for augmented reality, including developing improved tracking, display, and input technologies; creating tools for authoring AR content; developing novel AR applications and interaction techniques; and exploring new types of AR experiences through user evaluations. It also examines specific challenges like occlusion handling in see-through displays and examples of gesture and multimodal interaction research.

1. COSC 426: Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
Sept 19th 2012
Lecture 9: AR Research Directions

2. Looking to the Future

3. The Future is with us
It takes at least 20 years for new
technologies to go from the lab to the
lounge..
“The technologies that will significantly affect
our lives over the next 10 years have been
around for a decade.
The future is with us.The trick is learning how to
spot it.The commercialization of research,
in other words, is far more about Oct 11th 2004
prospecting than alchemy.”
Bill Buxton

4. Research Directions
experiences
Usability
applications Interaction
tools Authoring
components Tracking, Display
Sony CSL © 2004

5. Research Directions
 Components
 Markerless tracking, hybrid tracking
 Displays, input devices
 Tools
 Authoring tools, user generated content
 Applications
 Interaction techniques/metaphors
 Experiences
 User evaluation, novel AR/MR experiences

6. HMD Design

7. Occlusion with See-through HMD
 The Problem
 Occluding real objects with virtual
 Occluding virtual objects with real
Real Scene Current See-through HMD

8. ELMO (Kiyokawa 2001)
 Occlusive see-through HMD
 Masking LCD
 Real time range finding

9. ELMO Demo

10. ELMO Design
Virtual images
from LCD
Depth
Sensing LCD Mask
Real
World
Optical
Combiner
 Use LCD mask to block real world
 Depth sensing for occluding virtual images

11. ELMO Results

12. Future Displays
 Always on, unobtrusive

13. Google Glasses

14. Contact Lens Display
 Babak Parviz
 University Washington
 MEMS components
 Transparent elements
 Micro-sensors
 Challenges
 Miniaturization
 Assembly
 Eye-safe

15. Contact Lens Prototype

16. Applications

17. Interaction Techniques
 Input techniques
 3D vs. 2D input
 Pen/buttons/gestures
 Natural Interaction
 Speech + gesture input
 Intelligent Interfaces
 Artificial agents
 Context sensing

18. Flexible Displays
 Flexible Lens Surface
 Bimanual interaction
 Digital paper analogy
Red Planet, 2000

19. Sony CSL © 2004

20. Sony CSL © 2004

21. Tangible User Interfaces (TUIs)
 GUMMI bendable display prototype
 Reproduced by permission of Sony CSL

22. Sony CSL © 2004

23. Sony CSL © 2004

24. Lucid Touch
 Microsoft Research & Mitsubishi Electric Research Labs
 Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J., Shen, C.
LucidTouch: A See-Through Mobile Device
In Proceedings of UIST 2007, Newport, Rhode Island, October 7-10, 2007,
pp. 269–278.

26. Auditory Modalities
 Auditory
 auditory icons
 earcons
 speech synthesis/recognition
 Nomadic Radio (Sawhney)
- combines spatialized audio
- auditory cues
- speech synthesis/recognition

27. Gestural interfaces
 1. Micro-gestures
 (unistroke, smartPad)
 2. Device-based gestures
 (tilt based examples)
 3. Embodied interaction
 (eye toy)

28. Natural Gesture Interaction on Mobile
 Use mobile camera for hand tracking
 Fingertip detection

29. Evaluation
 Gesture input more than twice as slow as touch
 No difference in naturalness

30. Haptic Modalities
 Haptic interfaces
 Simple uses in mobiles? (vibration instead of ringtone)
 Sony’s Touchengine
- physiological experiments show you can perceive two stimulus 5ms
apart, and spaced as low as 0.2 microns
4 µｍ n層
28 µｍ
n層
V

31. Haptic Input
 AR Haptic Workbench
 CSIRO 2003 – Adcock et. al.

32. AR Haptic Interface
 Phantom, ARToolKit, Magellan

33. Natural Interaction

34. The Vision of AR

35. To Make the Vision Real..
 Hardware/software requirements
 Contact lens displays
 Free space hand/body tracking
 Environment recognition
 Speech/gesture recognition
 Etc..

36. Natural Interaction
 Automatically detecting real environment
 Environmental awareness
 Physically based interaction
 Gesture Input
 Free-hand interaction
 Multimodal Input
 Speech and gesture interaction
 Implicit rather than Explicit interaction

37. Environmental Awareness

38. AR MicroMachines
 AR experience with environment awareness
and physically-based interaction
 Based on MS Kinect RGB-D sensor
 Augmented environment supports
 occlusion, shadows
 physically-based interaction between real and
virtual objects

39. Operating Environment

40. Architecture
 Our framework uses five libraries:
 OpenNI
 OpenCV
 OPIRA
 Bullet Physics
 OpenSceneGraph

41. System Flow
 The system flow consists of three sections:
 Image Processing and Marker Tracking
 Physics Simulation
 Rendering

42. Physics Simulation
 Create virtual mesh over real world
 Update at 10 fps – can move real objects
 Use by physics engine for collision detection (virtual/real)
 Use by OpenScenegraph for occlusion and shadows

43. Rendering
Occlusion Shadows

44. Natural Gesture Interaction

45. Mo#va#on
AR
MicroMachines
and
PhobiAR
•
Treated
the
environment
as
sta/c
–
no
tracking
•
Tracked
objects
in
2D
More
realis#c
interac#on
requires
3D
gesture
tracking

46. Mo#va#on
Occlusion
Issues
AR
MicroMachines
only
achieved
realis/c
occlusion
because
the
user’s
viewpoint
matched
the
Kinect’s
Proper
occlusion
requires
a
more
complete
model
of
scene
objects

47. HITLabNZ’s Gesture Library
Architecture
5. Gesture
• Static Gestures
• Dynamic Gestures
• Context based Gestures
4. Modeling
• Hand recognition/modeling
• Rigid-body modeling
3. Classification/Tracking
2. Segmentation
1. Hardware Interface

48. HITLabNZ’s Gesture Library
Architecture
5. Gesture o Supports PCL, OpenNI, OpenCV, and Kinect SDK.
o Provides access to depth, RGB, XYZRGB.
• Static Gestures o Usage: Capturing color image, depth image and concatenated
• Dynamic Gestures point clouds from a single or multiple cameras
o For example:
• Context based Gestures
4. Modeling
• Hand recognition/
modeling Kinect for Xbox 360
• Rigid-body modeling
3. Classification/Tracking Kinect for Windows
2. Segmentation
Asus Xtion Pro Live
1. Hardware Interface

49. HITLabNZ’s Gesture Library
Architecture
5. Gesture o Segment images and point clouds based on color, depth and
space.
• Static Gestures o Usage: Segmenting images or point clouds using color
• Dynamic Gestures models, depth, or spatial properties such as location, shape
and size.
• Context based Gestures o For example:
4. Modeling
• Hand recognition/
modeling
• Rigid-body modeling Skin color segmentation
3. Classification/Tracking
2. Segmentation Depth threshold
1. Hardware Interface

50. HITLabNZ’s Gesture Library
Architecture
5. Gesture o Identify and track objects between frames based on
XYZRGB.
• Static Gestures o Usage: Identifying current position/orientation of the tracked
• Dynamic Gestures object in space.
• Context based Gestures o For example:
4. Modeling
• Hand recognition/ Training set of hand
modeling poses, colors
• Rigid-body modeling represent unique
regions of the hand.
3. Classification/Tracking
2. Segmentation Raw output (without-
cleaning) classified
on real hand input
1. Hardware Interface
(depth image).

51. HITLabNZ’s Gesture Library
Architecture
5. Gesture o Hand Recognition/Modeling
 Skeleton based (for low resolution
• Static Gestures approximation)
• Dynamic Gestures  Model based (for more accurate
• Context based Gestures representation)
o Object Modeling (identification and tracking rigid-
4. Modeling body objects)
• Hand recognition/ o Physical Modeling (physical interaction)
modeling  Sphere Proxy
• Rigid-body modeling  Model based
 Mesh based
3. Classification/Tracking o Usage: For general spatial interaction in AR/VR
environment
2. Segmentation
1. Hardware Interface

52. Method
Represent
models
as
collec#ons
of
spheres
moving
with
the
models
in
the
Bullet
physics
engine

53. Method
Render
AR
scene
with
OpenSceneGraph,
using
depth
map
for
occlusion
Shadows
yet
to
be
implemented

54. Results

55. HITLabNZ’s Gesture Library
Architecture
5. Gesture o Static (hand pose recognition)
o Dynamic (meaningful movement recognition)
• Static Gestures o Context-based gesture recognition (gestures with context,
• Dynamic Gestures e.g. pointing)
o Usage: Issuing commands/anticipating user intention and high
• Context based Gestures level interaction.
4. Modeling
• Hand recognition/
modeling
• Rigid-body modeling
3. Classification/Tracking
2. Segmentation
1. Hardware Interface

56. Multimodal Interaction

57. Multimodal Interaction
 Combined speech input
 Gesture and Speech complimentary
 Speech
- modal commands, quantities
 Gesture
- selection, motion, qualities
 Previous work found multimodal interfaces
intuitive for 2D/3D graphics interaction

58. 1. Marker Based Multimodal Interface
 Add speech recognition to VOMAR
 Paddle + speech commands

60. Commands Recognized
 Create Command "Make a blue chair": to create a virtual
object and place it on the paddle.
 Duplicate Command "Copy this": to duplicate a virtual object
and place it on the paddle.
 Grab Command "Grab table": to select a virtual object and
place it on the paddle.
 Place Command "Place here": to place the attached object in
the workspace.
 Move Command "Move the couch": to attach a virtual object
in the workspace to the paddle so that it follows the paddle
movement.

61. System Architecture

62. Object Relationships
"Put chair behind the table”
Where is behind?
View specific regions

63. User Evaluation
 Performance time
 Speech + static paddle significantly faster
 Gesture-only condition less accurate for position/orientation
 Users preferred speech + paddle input

64. Subjective Surveys

65. 2. Free Hand Multimodal Input
 Use free hand to interact with AR content
 Recognize simple gestures
 No marker tracking
Point Move Pick/Drop

66. Multimodal Architecture

67. Multimodal Fusion

68. Hand Occlusion

69. User Evaluation
 Change object shape, colour and position
 Conditions
 Speech only, gesture only, multimodal
 Measure
 performance time, error, subjective survey

70. Experimental Setup
Change object shape
and colour

71. Results
 Average performance time (MMI, speech fastest)
 Gesture: 15.44s
 Speech: 12.38s
 Multimodal: 11.78s
 No difference in user errors
 User subjective survey
 Q1: How natural was it to manipulate the object?
- MMI, speech significantly better
 70% preferred MMI, 25% speech only, 5% gesture only

72. Intelligent Interfaces

73. Intelligent Interfaces
 Most AR systems stupid
 Don’t recognize user behaviour
 Don’t provide feedback
 Don’t adapt to user
 Especially important for training
 Scaffolded learning
 Moving beyond check-lists of actions

74. Intelligent Interfaces
 AR interface + intelligent tutoring system
 ASPIRE constraint based system (from UC)
 Constraints
- relevance cond., satisfaction cond., feedback

75. Domain Ontology

76. Intelligent Feedback
 Actively monitors user behaviour
 Implicit vs. explicit interaction
 Provides corrective feedback

78. Evaluation Results
 16 subjects, with and without ITS
 Improved task completion
 Improved learning

79. Intelligent Agents
 AR characters
 Virtual embodiment of system
 Multimodal input/output
 Examples
 AR Lego, Welbo, etc
 Mr Virtuoso
- AR character more real, more fun
- On-screen 3D and AR similar in usefulness

80. Context Sensing

81. Context Sensing
 TKK Project
 Using context to
manage information
 Context from
 Speech
 Gaze
 Real world
 AR Display

85. Gaze Interaction

86. AR View

87. More Information Over Time

88. Experiences

89. Novel Experiences
 Crossing Boundaries
 Ubiquitous VR/AR
 Collaborative Experiences
 Massive AR
 AR + Social Networking
 Usability

90. Crossing Boundaries
Jun Rekimoto, Sony CSL

91. Invisible Interfaces
Jun Rekimoto, Sony CSL

92. Milgram’s Reality-Virtuality continuum
Mixed Reality
Real Augmented Augmented Virtual
Environment Reality (AR) Virtuality (AV) Environment
Reality - Virtuality (RV) Continuum

93. The MagicBook
Reality Augmented Augmented Virtuality
Reality (AR) Virtuality (AV)

94. Invisible Interfaces
Jun Rekimoto, Sony CSL

95. Example: Visualizing Sensor Networks
 Rauhala et. al. 2007 (Linkoping)
 Network of Humidity Sensors
 ZigBee wireless communication
 Use Mobile AR to Visualize Humidity

98. Invisible Interfaces
Jun Rekimoto, Sony CSL

99. UbiVR – CAMAR
CAMAR Controller
CAMAR Viewer
CAMAR Companion
GIST - Korea

100. ubiHome @ GIST
Media services Light service MR window
ubiTrack
Where/When
Tag-it
ubiKey
©ubiHome
Who/What/
What/When/How When/How
PDA Couch Sensor Door Sensor
Who/What/When/How When/How When/How

101. CAMAR - GIST
(CAMAR: Context-Aware Mobile
Augmented Reality)

102.  UCAM: Architecture wear-UCAM
Content
Sensor
Service
(Integrator,Manager,
Interpreter,ServiceProvider)
Context Interface
Network Interface
ubi-UCAM
BAN/PAN TCP/IP
(BT) (Discovery,Control,Event)
Operating System
vr-UCAM

103. Hybrid User Inerfaces
Goal: To incorporate AR into normal meeting
environment
 Physical Components
 Real props
 Display Elements
 2D and 3D (AR) displays
 Interaction Metaphor
 Use multiple tools – each relevant for the task

104. Hybrid User Interfaces
1 2 3 4
PERSONAL TABLETOP WHITEBOARD MULTIGROUP
Private Display Private Display Private Display Private Display
Group Display Public Display Group Display
Public Display

105. Ubiquitous
UbiComp
Ubi AR
Ubi VR
Weiser Mobile AR
Desktop AR VR
Terminal
Reality Virtual Reality
Milgram
From: Joe Newman

106. Massive
Multi User
Ubiquitous
r
Weise
Terminal
Single User
Reality
Milg
ram
VR

107. Remote Collaboration

108. AR Client
 HMD and HHD
 Showing virtual images over real world
 Images drawn by remote expert
 Local interaction

109. Shared Visual Context (Fussell ,1999)
 Remote video collaboration
 Shared manual, video viewing
 Compared Video, Audio, Side-by-side collaboration
 Communication analysis

110. WACL(Kurata,2004)
 Wearable Camera/Laser Pointer
 Independent pointer control
 Remote panorama view

111. WACL(Kurata,2004)
 Remote Expert View
 Panorama viewing, annotation, image capture

112. As If Being There (Poelman, 2012)
 AR + Scene Capture
 HMD viewing, remote expert
 Gesture input
 Scene capture (PTAM), stereo camera

113. As If Being There (Poelman, 2012)
 Gesture Interaction
 Hand postures recognized
 Menu superimposed on hands

114. Real World Capture
 Using Kinect for 3D Scene Capture
 Camera tracking
 AR overlay
 Remote situational awareness

115. Remote scene capture with AR annotations added

116. Future Directions
SLIDE 116
Massive Multiuser
 Handheld AR for the first time allows extremely high
numbers of AR users
 Requires
 New types of applications/games
 New infrastructure (server/client/peer-to-peer)
 Content distribution…

117. Massive MultiUser
 2D Applications
 MSN – 29 million
 Skype – 10 million
 Facebook – 100m+
 3D/VR Applications
 SecondLife > 50K
 Stereo projection - <500
 Augmented Reality
 Shared Space (1999) - 4
 Invisible Train (2004) - 8

118. BASIC VIEW

119. PERSONAL VIEW

120. Augmented Reality 2.0 Infrastructure

121. Leveraging Web 2.0
 Content retrieval using HTTP
 XML encoded meta information
 KML placemarks + extensions
 Queries
 Based on location (from GPS, image recognition)
 Based on situation (barcode markers)
 Queries also deliver tracking feature databases
 Everybody can set up an AR 2.0 server
 Syndication:
 Community servers for end-user content
 Tagging
 AR client subscribes to arbitrary number of feeds

122. Content
 Content creation and delivery
 Content creation pipeline
 Delivering previously unknown content
 Streaming of
 Data (objects, multi-media)
 Applications
 Distribution
 How do users learn about all that content?
 How do they access it?

123. ARML (AR Markup Language)

124. Scaling Up
 AR on a City Scale
 Using mobile phone as ubiquitous sensor
 MIT Senseable City Lab
 http://senseable.mit.edu/

126. WikiCity Rome (Senseable City Lab MIT)

127. Conclusions

128. AR Research in the HIT Lab NZ
 Gesture interaction
 Gesture library
 Multimodal interaction
 Collaborative speech/gesture interfaces
 Mobile AR interfaces
 Outdoor AR, interaction methods, navigation tools
 AR authoring tools
 Visual programming for AR
 Remote Collaboration
 Mobile AR for remote interaction

129. More Information
• Mark Billinghurst
– mark.billinghurst@hitlabnz.org
• Websites
– http://www.hitlabnz.org/
– http://artoolkit.sourceforge.net/
– http://www.osgart.org/
– http://www.hitlabnz.org/wiki/buildAR/