Mobile Augmented Reality
Kari Pulli
Research Fellow
Nokia Research Center Palo Alto
26 August, 2009 Kari Pulli | NRC © 2008 1

Imagine a world where anything
can be considered interactive
interactive…
Weather
Hotel
Information
Shopping
Restaurant
reviews & menus Call a taxi
26 August, 2009 Kari Pulli | NRC © 2008 2

MARA
Sensors
· GPS
· compass
· accelerometers
Camera
· used to give context over
which icons are overlaid
Robust, building-level accuracy
0:45 – 1:40
26 August, 2009 Kari Pulli | NRC © 2008 3

Mobile Augmented Reality
26 August, 2009 Kari Pulli | NRC © 2008 4

Image Space
A user-created mirror world
• reflecting th physical world
fl ti the h i l ld
• intuitive spatial browsing
experience
p
Representation
• Sparse representation of the world Nokia Image Space and Augmented
Reality at Mobile World Congress 2009
based on geo-registered images,
audio and video
• 3D reconstruction or models of Introduced in Nokia
buildings, etc., based on freely- Beta Labs in P6/09
taken images
26 August, 2009 Kari Pulli | NRC © 2008 5

Intuitive Navigation
Meaningful navigation through the physical
world by following image story-paths –
y g g yp
user generated interactions
You wonder which part of the city to
explore next. Turn your device to see
spatially interlinked images taken by
others to experience alternative routes
through the city.
© 2008 Nokia 6
V1-Filename.ppt /

MAR: Using images to find out what you’re pointing at
From an image… …to information
26 August, 2009 Kari Pulli | NRC © 2008 7

System Overview
GPS
Server
0:30-1:40
G. Takacs, V. Chandrasekhar, N. Gelfand, Y. Xiong, W-C Chen, T. Bismpigiannis, R. Grzeszczuk, K. Pulli, B. Girod
Outdoor Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
26 August, 2009 Kari Pulli | NRC © 2008 8
ACM International Conference on Multimedia Information Retrieval (MIR'08)

Landmark-based navigation
9

Good landmarks
Image count is an indicator of landmark popularity
Require good visibility f
R i d i ibilit from current l
t location
ti
Prefer landmarks that are straight ahead
10

Augment images
Use known locations to estimate camera direction
Draw an arrow i th i
D in the image
Generate relative text directions
Keep walking straight, toward West Gate
H. Hile, R. Grzeszczuk A. Liu, R. Vedantham, J. Kosecka, G. Borriello
Landmark-based Pedestrian navigation with Enhanced Spatial Reasoning
11
Pervasive 2009.

Goal: Bring AR to outdoor environment
Location-based context
information
 Need: Large-scale scene recognition
 Must use scale-invariant features
with strong descriptors for matching
Target mobile devices
 Need: Efficient real-time tracking on
the mobile platform
 Cannot detect and match scale-invariant
features for every frame!
Duy-Nguyen Ta, Natasha Gelfand, Wei-Chao Chen, Kari Pulli
12
SURFTrac: Efficient Tracking and Continuous Object Recognition using Local Feature Descriptors
IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'09)

Matching and Tracking with an Image Database
13

Nokia Point & Find: Point, Browse, Get What You Want Now!
26 August, 2009 Kari Pulli | NRC © 2008 14

Product Vision: Nokia Point & Find
Bridges The Physical World With The Digital World
Physical World Digital World
15
Future release
15 © 2008 Nokia

Point of sale example: point at products in store and
g
get details, p
promos, videos, more
Shopping in a Instant Results:
store...
 Detailed product
info
Point phone
 Video demo
at a product
(camera)  Virtual tour
 Special offers
(coupon or rebate)
 Accessories
 Send message to
friend about product
p
16
16 © 2008 Nokia

Interaction metaphors
Previous projects use the “Hand-held Magic Lens” metaphor
· camera on th “f side” of the d i grabs i
the “far id ” f th device b image
· device adds annotations
· display on the “near side of the device
near side”
What about data goggles?
· that’s what Augmented / Mixed Reality used to be all about
· is that feasible?
26 August, 2009 Kari Pulli | NRC © 2008 17

Prototype – Near-Eye-Display
Display
Make it thin with and optics
diffractive optics
Output Gratings Stacked EPE (2x) Output Gratings
Input Grating
26 August, 2009 Kari Pulli | NRC © 2008 18

Prototype – Near-Eye Gaze Tracking
Display
and optics
Camera
Output
Output Gratings Stacked EPE (3x) Gratings
Input Grating
26 August, 2009 Kari Pulli | NRC © 2008 19

Prototype
26 August, 2009 Kari Pulli | NRC © 2008 20

Interaction via gaze – Possible applications
0:00 – 0:25
• Pointing
• Eye typing
E t i
• Image / video
p
panning / zooming
g g
• User monitoring
• …
26 August, 2009 Kari Pulli | NRC © 2008 21

Summary
Blended realities – Why?
· access i f
information visually, on l
ti i ll location
ti
Main metaphor
· magic lens: annotate view of real world
Devices
· hand-held camera phone NOW
· data glasses FUTURE
26 August, 2009 Kari Pulli | NRC © 2008 22