• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
426 Lecture 8: Mobile Augmented Reality

426 Lecture 8: Mobile Augmented Reality



COSC 426 graduate class on Mobile Augmented Reality taught at the University of Canterbury by Mark Billinghurst, Sept. 12th 2012.

COSC 426 graduate class on Mobile Augmented Reality taught at the University of Canterbury by Mark Billinghurst, Sept. 12th 2012.



Total Views
Views on SlideShare
Embed Views



1 Embed 553

http://augmentedrealitybiz.com 553



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    426 Lecture 8: Mobile Augmented Reality 426 Lecture 8: Mobile Augmented Reality Presentation Transcript

    • COSC 426: Augmented Reality Mark Billinghurst mark.billinghurst@hitlabnz.org Sept 12th 2012 Lecture 8: Mobile Augmented Reality
    • 1983 – Star Wars
    • 1999 - HIT Lab US
    • 1998: SGI O2 2008: Nokia N95CPU: 300 Mhz CPU: 332 MhzHDD; 9GB HDD; 8GBRAM: 512 mb RAM: 128 mbCamera: VGA 30fps Camera: VGA 30 fpsGraphics: 500K poly/sec Graphics: 2m poly/sec
    • Mobile Phone AR  Mobile Phones   camera   processor   display  AR on Mobile Phones   Simple graphics   Optimized computer vision   Collaborative Interaction
    • 2005: Collaborative AR  AR Tennis   Shared AR content   Two user game   Audio + haptic feedback   Bluetooth networking
    • Mobile AR History
    • Evolution of Mobile AR Camera phone Camera phone - Thin client ARWearable Wearable ARComputers Camera phone - Self contained AR PDAs Handheld -Thin client AR AR Displays PDAs -Self contained AR 1995 1997 2001 2003 2004
    • Handheld DisplaysTethered Applications   Fitzmaurice Chameleon (1994)   Rekimoto’s Transvision (1995)   Tethered LCD   PC Processing and Tracking
    • Handheld AR Display - Tethered1995, 1996 Handheld AR   ARPad, Cameleon   Rekimoto’s NaviCam, Transvision   Tethered LCD   PC Processing and Tracking
    • NaviCam (Rekimoto, 1995)Information is registered to real-world context   Hand held AR displaysInteraction   Manipulation of a window into information spaceApplications   Context-aware information displays
    • AR Pad (Mogilev 2002)Handheld AR Display   LCD screen   Camera   SpaceOrb 3 DOF controller   Peripheral awareness   Viewpoint awareness
    • Mobile AR: Touring Machine (1997)  University of Columbia   Feiner, MacIntyre, Höllerer, Webster  Combines   See through head mounted display   GPS tracking   Orientation sensor   Backpack PC (custom)   Tablet input
    • MARS View  Virtual tags overlaid on the real world  “Information in place”
    • Backpack/Wearable AR1997 Backpack AR   Feiner’s Touring Machine   AR Quake (Thomas)   Tinmith (Piekarski)   MCAR (Reitmayr)   Bulky, HMD based
    • Mobile AR - Hardware GPS Example self-built working Antenna solution with PCI-based 3D graphics PCI 3D Graphics Board Tracker Controller PC104 Sound Card DC to DC Wearable Converter CPU Computer PC104 PCMCIA Battery GPS RTK Hard Drive correction Radio Serial PortsColumbia Touring Machine
    • Sharp J-SH04  1997 Philip Kahn invents camera phone  1999 First commercial camera phone
    • Millions of Camera Phones
    • Handheld AR – Thin Client2001 BatPortal (AT&T Cambridge)   PDA used as I/O device   Wireless connection to workstation   Room-scale ultrasonic tracking (Bat)2001 AR-PDA (C Lab)   PDA thin graphics client   Remote image processing   www.ar-pda.com
    • Mobile Phone AR – Thin Client2003 ARphone (Univ. of Sydney)   Transfer images via Bluetooth (slow – 30 sec/image)   Remote processing – AR Server  
    • Early Phone Computer Vision Apps2003 – Mozzies Game - Best mobile gameOptical motion flow detecting phone orientationSiemens SX1 – Symbian, 120Mhz, VGA Camera2005 – Marble Revolution (Bit-Side GmbH)Winner of Nokias Series 60 Challenge 20052005 – SymBall (VTT)
    • Computer Vision on Mobile Phone  Cameras and Phone CPU sufficient for computer vision applications  Pattern Recognition (Static Processing)   QR Code   Shotcode (http://www.shotcode.com/)  Motion Flow (2D Image Processing)   GestureTek -  http://www.gesturetekmobile.com/   TinyMotion  3D Pose Calculation   Augmented Reality
    • Handheld AR – Self Contained2003 PDA-based AR   ARToolKit port to PDA   Studierstube ported to PDA   AR Kanji Educational App.   Mr Virtuoso AR character   Wagner’s Invisible Train -  Collaborative AR
    • Mobile Phone AR – Self Contained2004 Mobile Phone AR   Moehring, Bimber   Henrysson (ARToolKit)   Camera, processor, display together
    • AR Advertising  Txt message to download AR application (200K)  See virtual content popping out of real paper advert  Tested May 2007 by Saatchi and Saatchi
    • 2008 - Location Aware PhonesMotorola Droid Nokia Navigator
    • Real World Information Overlay  Tag real world locations   GPS + Compass input   Overlay graphics data on live video  Applications   Travel guide, Advertising, etc  Eg: Mobilizy Wikitude (www.mobilizy.com)   Android based, Public API released  Other companies   Layar, AcrossAir, Tochnidot, RobotVision, etc
    • Layar – www.layar.com
    • HIT Lab NZ Android AR Platform  Architectural Application  Loads 3D models   a OBJ/MTL format  Positions content in space   GPS, compass  Intuitive user interface   toolkit to modify the model  Connects to back end model database
    • Architecture Web Interface Add models Web application java and php server Android application Database server Postgres
    • History of Handheld and Mobile AR1995 Handheld Display: NaviCam, AR-PAD, Transvision1997 Wearable AR: Touring Machine, AR Quake2001 Handheld AR – Thin Client: AR-PDA, Bat Portal2003 Handheld AR – Self contained: Invisible Train2003 Mobile Phone – 2D Vision: Mozzies, Symball2003 Mobile Phone – Thin Client: ARphone2004 Mobile Phone – Self contained: Moehring, Symbian
    • 2012 State of the ArtHandheld Hardware available PDA, mobile phones, external cameras Sensors: GPS, accelerometer, compassSoftware Tools are Available Tracking: ARToolKitPlus, stbTracker, Vuforia Graphics: OpenGL ES Authoring: Layar, Wikitude, Metaio CreatorWhat is needed: High level authoring tools Content development tools Novel interaction techniques User evaluation and usability
    • Mobile AR Companies  Mobile AR   GPS + compass  Many Companies   Layar   Wikitude   Acrossair   PressLite   Yelp   Robot vision   Etc..
    • $2 million USD in 2010$732 million USD in 2014
    • Handset Manufacturers  Qualcomm   $100 million USD investment  Nokia   25+ people in NRC  Samsung   Exploring the space  Apple   586 AR Applications on App Store  Google   Google goggles/Android AR Applications
    • Qualcomm  Acquired Imagination  October 2010 - Releases free Android AR SDK  Computer vision tracking - marker, markerless  Integrated with Unity 3D renderer  http://developer.qualcomm.com/ar
    • Rock-em Sock-em  Shared AR Demo  Markerless tracking
    • Apple  iPhone 4 SDK supports direct camera access  Launches AR theme on App Store
    • Mobile AR Technology
    • Technology Components  Software Platform   Eg studierStube platform  Developer Tools   iOS, Android  Tracking Technology   Computer vision, sensor based  Mobile Graphics   OpenGL ES  Interaction Methods   Handheld Interaction
    • What are the challenges?  Many platforms (operating systems)  Slow CPUs (low clock rates, often no FPU)   Difficulties with tracking   Difficulties with visualizations that require a lot of data processing  Small and slow memory access  No or weak hardware 3D acceleration   Difficulties with graphics  Poor cameras   Bad under low light, noise, etc
    • What makes a device interesting for AR?  Open and easy to program  Good camera  Fast CPU, FPU is a plus  Good H/W 3D support  Large installed basis   Easy access to devices  GPS, accelerometer, compass  Enough memory/storage
    • Typical Smart Phone Hardware  CPU   300-800+ Mhz  GPU   None, or Power VR Chip (OpenGL ES1.0/2.0)  Input   Touch screen, keyboard, keypad  Sensors   GPU, compass, accelerometer, camera (1.3-5mb+)  Networking   Bluetooth, Wifi, 3G  Screen   320x240 up to 800x480
    • HTC Desire  Android  Fast CPU (1GHz)  Smaller screen   3.7”, 800x480  Multi-touch  Hardware 3D  GPS, compass and accelerometer
    • iPhone 4  Apple iOS 4.0  Faster CPU (1.2GHz)  High screen resolution   3.5”, 960x640  (Finally) camera API  Multi-touch  Hardware 3D  GPS, compass, accelerometer and gyroscope
    • Hardware Sensors  Camera (resolution, fps)   Maker based/markerless tracking   Video overlap  GPS (resolution, update rate)   Outdoor location  Compass   Indoor/outdoor orientation  Accelerometer   Motion sensing, relative tilt
    • Studierstube ES Framework  Typical AR application framework  Developed at TU Graz
    • End UserApplicationProgrammingLibrariesOS/Low Level APIHardware
    • The Studierstube ES framework User Interface - Application Content Graphics Tracking Platform
    • Mobile Development Environments  Not like developing for desktops   Wide range of different OS   Limited CPU, low memory, poor graphics, no floating point  Popular Mobile OS   iPhone (Objective C)   Android (Java, Native NDK wrapper)   Windows Mobile (C#, C++, Visual Studio)  Other   Blackberry, Linux
    • Programming iPhone  Harsh restrictions from Apple   Apps have to go through the apps store  Xcode IDE for development   Nice development tools  Objective-C   Required for application development   Can call into C/C++ code  Limited camera API except in iOS 4   Can overlay on live video   No image processing
    • Android  Hardware creators   HTC, LG, Samsung, Motorola  Widely available phone   Different form factors – tablets, phones, PC, etc  Multiple versions and fragmentation   Android 1.0, 1.6   Android 2.0, 2.1  Free Tools   Eclipse Development   App Integrator
    • Mobile Graphics
    • Computer Graphics on Mobile Phones  Small screen, limited input options  Limited support for accelerated graphics   Most phones have no GPU  Mobile Graphics Libraries   OpenGL ES (1.0, 2.0) -  Cross platform, subset of OpenGL -  C/C++ low level library   Java M3G -  Mobile 3D graphics API for J2ME platform -  Object importer, scene graph library -  Support from all major phone manufacturers
    • OpenGL ES  Small-footprint subset of OpenGL   OpenGL is too large for embedded devices!  Powerful, low-level API, full functionality for 3D games   Can do almost everything OpenGL can   Available on all key platforms   Software and hardware implementations available  Fully extensible   Extensions like in OpenGL
    • OpenGL ES SLIDE 61OpenGL ES on mobile devices
    • OpenGL ES History  Khronos Group – founded 2000   Goal: create royalty free open multimedia standards   Create OpenML specification   Consortium of > 120 companies  OpenGL ES working group – formed 2002   20-30 companies   Create API for 3D graphics on embedded devices   Cleaned up and trimmed down version of OpenGL   OpenGL ES 1.0 released 2004, 1.1 in 2005, 2.0 spec in 2007
    • Versions  Two major tracks   Not compatible, parallel rather than competitive  OpenGL ES 1.x   Fixed function pipeline   Suitable for software implementations   All 1.x are backwards compatible  OpenGL ES 2.x   Vertex and pixel shaders using GLSL ES   All 2.x will be backwards compatible
    • Fixed Function (1.x) h"p://www.khronos.org/opengles/2_X/  
    • Programmable (2.x) h"p://www.khronos.org/opengles/2_X/  
    • OpenGL ES 1.x vs 2.0
    • Hardware supporting OpenGL ES 2.x  Qualcomm SnapDragon  Texas Instruments OMAP3  NVIDIA Tegra (APX2500)  Samsung S3C6410  POWERVR SGX  Now becoming available in phones!  Typically come with OpenGL ES 1.x drivers too   The emulation on GLES 2.0 hardware can be slow!
    • Tracking
    • Mobile Augmented Reality’s goalCreate an affordable, massively multi-user, widespread platform © Tinmith, U. of South Australia
    • Tracking Requirements for AR on Phones  Fast and efficient  Form factor: light and robust  Track simultaneously   a large number of objects   by a large number of users  Requires little or no …   Device modification   Manual calibration (targeting non-technical users)   Instrumentation of the physical environment  Low costs
    • How to not do it… Ka-Ping Yee: Peephole Displays, CHI’03
    • Tracking on mobile phones  Vision-based tracking   Marker-based tracking   Model-based natural feature tracking   Natural feature tracking in unknown environments  Sensor tracking   GPS, inertial compass, gyroscope
    • Backpack-based Höllerer et al. (1999), Piekarski & Thomas al. (2001), Reitmayr & Schmalstieg (2003)  Laptop, HMD  Enhanced GPS (DGPS / RTK) + inertial sensor for viewpoint tracking  Hand tracking w/ fiducial markers
    • Tablet PC / UMPC-based Schall et al., 2006  Hybrid tracking on UMPC   Camera  fiducial marker tracking   When no marker in view  inertial sensor + UWB tracking
    • PDA-based 1.BatPortal (Newman et al., 2001) SHEEP (MacWilliams et al., 2003)  PDA as thin client (rendering &   Tracking by ART (external IR tracking on server + VNC) cameras + retroreflective target)  Ultrasonic tracking   Projection-based AR environ.
    • PDA-based 2. Signpost on PDA (Wagner & Schmalstieg, 2003)  First “truly” handheld AR platform: PDA + camera  Standalone, self-contained AR system  Optimized fiducial marker tracking library
    • History of non-AR Tracking on Phones (1)AR-PDA (Gausemeier et al., 2003) Kick Real (Paelke et al., 2004)  Model-based tracking   Edge detection of real foot + collision  PDA = thin client detection w/ virtual ball  tracking off-loaded to server   2D tracking and limited interaction  Not real-time (tailored to game)
    • History of non-AR Tracking on Phones (2)Mosquito Hunt (Siemens, 2003)Marble Revolution (BitSide, 2004)Pingis (VTT, 2006) TinyMotion (Wang et al., 2006)Game control w/ optical GUI control & input on cell phonesflow techniques w/ image differencing & block correlation
    • History of AR Tracking on Phones (1)  2003   ARToolKit on PDA   Wagner et at.  2004   3D Marker on Phone   Möhring et al.  2005   ARToolKit on Symbian   Henrysson et al.
    • History of AR Tracking on Phones (2)  2005   Visual Codes   Rohs et at.  2008   Advanced Marker Tracking   Wagner et al.  2008   Natural Feature Tracking   Wagner et al.
    • What can we do on today‘s mobile phones?  Typical specs   600+ MHz   ~5MB of available RAM   160x120 - 320x240 at 15-30 Hz camera  Possible to do   Marker tracking in 5-15ms   Natural feature tracking in 20-50ms
    • Other Mobile Sensors  Orientation   Compass  Relative movement/rotation   Accelerometer, gyroscope  Context   Audio, light sensor, proximity  Location   GPS, A-GPS, Wifi positioning, Cell tower triangulation
    • Handheld AR Interfaces
    • Handheld HCI  Consider your user  Follow good HCI principles  Adapt HCI guidelines for handhelds  Design to device constraints  Rapid prototyping  User evaluation
    • Sample Handheld AR Interfaces  Clean  Large Video View  Large Icons  Text Overlay
    • Handheld Display vs Fixed Display  Experiment comparing handheld moving, to handheld button input, small fixed display, desktop display, large plasma  Users performed (1) navigation task, (2) selection task  Moving handheld display provided greater perceived FOV, higher degree of presence, faster completion timeJ. Hwang, J. Jung, G. Kim. Hand-held Virtual Reality: A Feasibility Study. In proceedings of VRST 2006
    • Search Task Completion Time
    • FOV, Presence and Immersion
    • Outdoor AR: Limited Field of View
    • Possible solutions   Overview + Detail   spatial separation; two views   Focus + Context   merges both views into one view   Zooming   temporal separation
    • Zooming Views  TU Graz – HIT Lab NZ - collaboration   Zooming panorama   Zooming Map
    • Zooming AR interfacesContext  Compass   Zooming  Panorama   Zooming  Map    Interface Types   Compass (C)   Compass + Zooming Panorama (CP)   Compass + Zooming Map (CM)   Compass, Zooming Panorama, Zooming Map (CPM)
    • Twitter 360  www.twitter-360.com  iPhone application  See geo-located tweets in real world  Twitter.com supports geo tagging
    • Wikitude – www.mobilizy.com Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah
    • Information Filtering
    • Information Filtering  Information Filtering (Julier et al. ’00) •  Remove clutter by goal- and distance based filtering •  User’s task is route finding: Sniper and relevant buildings are displayed; objects, which are determined to be unnecessary, removed
    • HMD vs Handheld AR Interface Wearable AR HandHeld AR Output: Display Input & Output Input
    • Handheld Interface Metaphors  Tangible AR Lens Viewing   Look through screen into AR scene   Interact with screen to interact with AR content -  Eg Invisible Train  Tangible AR Lens Manipulation   Select AR object and attach to device   Use the motion of the device as input -  Eg AR Lego
    • Translation StudyConditions A: Object fixed to the phone (one handed) B: Button and keypad input C: Object fixed to the phone (bimanual) - one hand for rotating tracking pattern
    • Results – Translation•  9 subjects – within subject design•  Timing •  Tangible fastest •  twice as fast as keypad•  Survey •  Tangible easiest (Q1) •  Keypad most accurate (Q2) •  Tangible quickest (Q3) •  Tangible most enjoyable (Q4)•  Ranking B A C •  Tangible favored Rank 1.44 2.56 2.0
    • Rotation StudyConditions A: Arcball B: Keypad input for rotation about the object axis C: Object fixed to the phone (one handed) D: Object fixed to the phone (bimanual)
    • Results – Rotation•  Timing •  Keypad(B) and Arcball(A) fastest•  No significant survey differences A B C D Rank 3.0 2.3 2.4 2.2
    • TAT Augmented ID
    • Design GuidelinesApply handheld HCI guidelines for on-screen content - large buttons, little text input, etcDesign physical + virtual interface elementsPick appropriate interface metaphor - “handheld lens” approach using handheld motion - Tangible AR for AR overlayBuild prototypesContinuously evaluate application
    • Mobile AR Browsing
    • Junaio - www.junaio.com
    • QR Code Launch
    • Glue Tracking - Markerless  Search for “instant tracker”
    • Junaio Interface
    • Interface  List View, Map View, AR View
    • Architecture
    • Back-end Servers
    • Data Flow
    • Search.php<?xml version="1.0" encoding="UTF-8"?><results> <poi id="1" interactionfeedback="none"> <name><![CDATA[[Hotel Hello World]]]></name> <description><![CDATA[[This is a beautiful, family hotel and restaurant, just around the corner. Special Dinner and Rooms available.]]]></description> <l>37.776685,-122.422771,0</l> <mime-type>text/plain</mime-type> <icon>http://dev.junaio.com/publisherDownload/tutorial/icon_map.png</icon> <thumbnail>http://dev.junaio.com/publisherDownload/tutorial/thumb.jpg</thumbnail> <phone>555/1234567</phone> <homepage> http://www.hotelaroundthecorner.com </homepage> </poi></results>
    • Creating a New Channel
    • AR Outcome
    • Mime Tag Types  <mime-type>text/plain</mime-type>  Changing mime tag changes media loaded
    • AR Browsers  Commercial outdoor AR applications   Junaio, Layar, Wikitude, etc  All have their own language specifications   Wikitude – ARML   Junaio - XML  Need for common standard   Based on existing standards for geo-located content etc   Support for dynamic/interactive content   Easier to author mobile AR applications   Easy to render on AR browsers
    • BirdsView - http://www.birdsview.de/  Location Based CMS   Add content, publish to Layar or Junaio
    • BirdsView on Junaio
    • BirdsView on Junaio
    • Limitations  BirdsView Branding   Their logo, not yours  Limited POI Content   Images, Text, URL  Public Channel   Your content + everyone elses
    • Hoppala Augmentation  http://www.hoppala-agency.com/  Rich media overlay
    • Hoppala in Junaio
    • Metaio Creator  Drag and drop Junaio authoring
    • BuildAR – buildar.com
    • 3DOn  Onsite Visualization Single Building   GPS + Compass input   Overlay graphics data on live video/Photos
    • 3D On Interface