Your SlideShare is downloading. ×

COSC 426 Lect. 5 - Mobile AR


Published on

Lecture 5 of the COSC 426 graduate course on Augmented Reality. This lecture provides an overview of Mobile Augmented Reality. The Lecture is given by Mark Billinghurst of the HIT Lab NZ at the …

Lecture 5 of the COSC 426 graduate course on Augmented Reality. This lecture provides an overview of Mobile Augmented Reality. The Lecture is given by Mark Billinghurst of the HIT Lab NZ at the University of Canterbury

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Lecture 5: Mobile AR Mark Billinghurst k billi h t@hitl b HIT Lab NZ University of Canterbury
  • 2. AR on mobile phonesLow cost, widely spread platform Billions of phones deployed People know how to use them Strong demand from commercial side Huge chance for AR!Target practical applications Easy to use Quality graphics Robust tracking 15-30 Hz overall frame rate
  • 3. Why would you use phones?Robust and fool-proof People know how to use their mobile phones…Variety of supported devicesSelf containedSelf-contained operation Enough processing power to do everything nativelyUltra-mobileUltra mobileLow cost devices Even better: people already own the target hardware! !A very unique chance for bringing AR to the masses
  • 4. Why would you not use phones?Compared to PC-based setups Less processing power and memory p gp y Harder to program, debug and deployHardware difficult or impossible to extendSmall number of available librariesTypically l littleT i ll only li l experience i research groups i in hSo many different devices and operating systems
  • 5. Other limitations of handheld ARUsual limitations in mobile HCI Small screen - Less information possible to display - Less immersion Limited input …Other limitations You see through the camera and not through the phone Switch attention between background and phone Strain factor of holding the phone up Social issues with pointing the phone at people p g p p p
  • 6. Mobile AR History
  • 7. Evolution of Mobile AR E l fM bl Camera phone Camera phone - Thin client ARWearable Wearable ARComputers Camera phone - Self contained AR PDAs Handheld -Thin client AR AR Displays p y PDAs -Self contained AR 1995 1997 2001 2003 2004
  • 8. Handheld DisplaysTethered Applications Fitzmaurice Chameleon (1994) ( ) Rekimoto’s Transvision (1995) Tethered LCD PC Processing and Tracking
  • 9. Handheld AR Display - Tethered1995, 1996 Handheld AR ARPad, ARPad Cameleon Rekimoto’s NaviCam, Transvision Tethered LCD PC Processing and Tracking
  • 10. Mobile AR: Touring Machine (1997)Columbia University Feiner, MacIntyre, Höllerer, WebsterCombines See through head mounted display GPS t ki tracking Orientation sensor Backpack PC ( p (custom) ) Tablet input
  • 11. MARS ViewVirtual tags overlaid on th real worldVi t l t l id the l ld“Information in place”
  • 12. Backpack/Wearable AR1997 Backpack AR Feiner’s Touring Machine AR Quake (Thomas) Tinmith (Piekarski) MCAR (Reitmayr) Bulky, HMD based
  • 13. Mobile AR - Hardware RTK correction Antenna GPS Example self-built working Antenna solution with PCI-based 3D graphics PCI based HMD Controller PCI 3D Graphics Board Tracker Controller PC104 Sound Card DC to DC Wearable Converter CPU Computer PC104 PCMCIA Battery GPS RTK Hard Drive correction Radio Serial PortsColumbia Touring Machine
  • 14. Sharp J-SH041997 Philip Kahn invents camera phone1999 First commercial camera phone
  • 15. Millions of Camera Phones12001000 800 DSC 600 Phone 400 200 0 2002 2003 2004 2005 2006 2007 2008 2009 2010
  • 16. Handheld AR – Thin Client2001 BatPortal (AT&T Cambridge) PDA used as I/O device Wireless connection to workstation Wi l i k i Room-scale ultrasonic tracking (Bat)2001 AR-PDA (C Lab) PDA thin graphics client Remote image processing
  • 17. Mobile Phone AR – Thin Client2003 ARphone (Univ. of Sydney) Transfer images via Bluetooth (slow – 30 sec/image) g ( g ) Remote processing – AR Server
  • 18. Early Phone Computer Vision Apps2003 – Mozzies Game - Best mobile gameOptical motion flow detecting phone orientationSiemens SX1 – S bi 120Mh VGA CameraSi Symbian, 120Mhz, C2005 – Marble Revolution (Bit-Side GmbH)Winner of Nokias Series 60 Challenge 2005Wi f N ki S i Ch ll2005 – SymBall (VTT)
  • 19. Computer Vision on Mobile PhoneCameras and Phone CPU sufficient for computer visionapplicationsPattern Recognition (Static Processing) g ( g) QR Code Shotcode ( Flow (2D IM i Fl Image P Processing) i ) GestureTek - TinyMotion3D Pose Calculation Augmented Reality
  • 20. Handheld AR – Self Contained2003 PDA-based AR ARToolKit port to PDA Studierstube ported to PDA AR Kanji Educational App. Mr Virtuoso AR character Wagner’s Invisible Train - Collaborative AR
  • 21. Mobile Phone AR – Self Contained2004 M bil Phone AR Mobile Ph Moehring, Bimber Henrysson (ARToolKit) Camera, processor, display together
  • 22. Location Aware PhonesMotorola Droid Nokia Navigator
  • 23. Real World Information OverlayTag real world locations GPS + Compass input Overlay graphics data on live videoApplications Travel guide, Advertising, etcEg: Mobilizy Wikitude ( Android based, Public API releasedOther companies Layar, AcrossAir, Tochnidot, RobotVision, etc
  • 24. Layar –
  • 25. HIT Lab NZ Android AR PlatformArchitectural ApplicationLoads 3D models a OBJ/MTL formatPositions content in space p GPS, compassIntuitive user interface toolkit to modify the modelConnects to back end model database
  • 26. HIT Lab NZ Mobile Outdoor AR
  • 27. History of Handheld and Mobile AR1995 Handheld Display: NaviCam, AR-PAD, Transvision1997 Wearable AR: Touring Machine, AR Quake g2001 Handheld AR – Thin Client: AR-PDA, Bat Portal2003 Handheld AR – Self contained: Invisible Train2003 Mobile Phone – 2D Vision: Mozzies, Symball2003 Mobile Phone – Thin Client: ARphone2004 Mobile Phone – Self contained: Moehring, Symbian
  • 28. 1996 Mobile AR by Weight 2003 2007 Scale it down: Scale it down more: Vesp R Vesp‘R [Kruijff ISMAR07]: Smartphone…$500 S t h $500 …Sony UMPC 1.1GHz …All-in-oneBackpack+HMD: …1.5kg …0.1kg…5-8kg 5 8k …still >$5K till …billions of units
  • 29. 2011 S State of the Art f h AHandheld Hardware available PDA, mobile phones, external cameras Sensors: GPS, accelerometer, compassSoftware Tools are Available Tracking: ARToolKitPlus, QCAR Graphics: OpenGL ES Authoring: Studierstube, Layar, Wikitude, UnifyeWhat is needed: High level authoring tools Content development tools Novel interaction techniques User evaluation and usability
  • 30. Mobile AR CompaniesMobile AR GPS + compassMany CompaniesM C i Layar Wikitude Acrossair PressLite Yelp Robot vision Etc.. E
  • 31. $2 million USD in 2010$732 million USD in 2014
  • 32. Handset ManufacturersQualcomm $100 million USD investmentNokiaN ki 25+ people in NRCSamsung Exploring the spaceApple 586 AR Applications on App StoreGoogle Google goggles/Android AR Applications g g gg pp
  • 33. QualcommAcquired Imagination q gOctober 2010 - Releases free Android AR SDKComputer vision tracking - marker, markerless p gIntegrated with Unity 3D renderer p p q
  • 34. Rock-em Sock-emSharedSh d AR D DemoMarkerless tracking
  • 35. AppleiPhone 4 SDK supports direct camera access ppLaunches AR theme on App Store
  • 36. > 500 AR apps on App Store
  • 37. Developing AR applications
  • 38. Mobile AR TechnologyInvolves Tracking g Content Loading Rendering/3D graphics g g p User Interface Application Design Evaluation
  • 39. Scientific challengesAR requires (unlike related disciplines) Strict real time operation (30Hz) - Unlike Ubicomp or mobile information systems p y High spatial precision (1cm, 1 degree) - Unlike location-based services Robustness for operation by human user - Unlike many computer vision methods in automation etc.Mobile hM bil phone AR requires (in addition) i (i ddi i ) No thin client! Same l l of performance as desktop AR S level f f d k - New algorithms must be orders of magnitude more efficient No unrealistic assumptions about HW
  • 40. How does a basic AR application work?Main loop Get a video frame from the camera Estimate the position and the orientation - computer vision, sensor input (GPS, compass) p p ( p ) Render the augmented scene (video + virtual content) ) Render GUI Process user input Update application status
  • 41. Studierstube ES FrameworkTypical AR application ApplicationsframeworkDeveloped at TU StudierstGraz Platformtube Software Stack a
  • 42. End UserApplication ppProgrammingLibrariesLib iOS/Low Level APIHardware
  • 43. The Studierstube ES frameworkTh S d b f k Applications o User Interface - Application Studie Content erstube Software S Graphics Stack Tracking Platform Platform
  • 44. Platform
  • 45. What are the challenges?Experiences with embedded platforms requiredMany platforms (operating systems)M l f ( i )Slow CPUs (low clock rates, often no FPU) Difficulties with tracking Difficulties with visualizations that require a lot of data processing iSlow memory accessNo or weak hardware 3D acceleration Difficulties with graphics
  • 46. Processing power of mobile phonesWeak processing power ~1GHz, ~1GHz Single core Often no floating point unit - Floating po t co e ~40x s owe t a integer co e oat g point code 0 slower than tege code - Fixed point problematic for many algorithms - Requires good math libraryCode optimized for phones runs 5-10x slower on ahigh-end phone than on an average PCNot going to change dramatically due to battery power
  • 47. So what are the common problems?Bad camera quality under low lighting Noise, motion blur , Strongly varies with different phonesSmall memory Keeping large databases in memory is problematicSlow memory Low clock rate - Processing large memory areas is prohibitive g g y p - Typical CV building blocks (SVD, image processing) are too demanding Slow data transfers between CPU and GPU
  • 48. Platforms for handheld AR Pl f f h dh ld Pros ConsWindows  Easy to program and debug Camera drivers are not always goodMobile Lots of devices Largest installed basis Hard to program and debug, SDK changes Symbian Good devices often, usually slow CPUs Very nice hardware Very nice hardware Camera API only with Camera API only with OS4iPhone Hype factor Objective C as main language Increasing number of devicesAndroid Java as main language Java as main language Hype factorLinux (LiMo,  Full Linux support, limited Large set of different libraries Large set of different librariesMaemo)M ) handsets  for AR h d f ARRIM  Widely spread in the US Java onlyBlackberryPalm  Only very few devices so far Nice hardwareWebOS No native SDK so far
  • 49. Worldwide smartphone market share 1Q09 1Q10 Symbian RIM iPhone Android Source: Gartner
  • 50. 2011 US Market Share
  • 51. What makes a device interesting for AR? Open and easy to program Good camera Fast CPU, FPU is a plus Good H/W 3D support Large installed basis g Easy access to devices GPS, accelerometer, compass Enough memory/storage
  • 52. Typical Smart Phone HardwareCPU 300-800+ MhzGPU None, or Power VR Chip (OpenGL ES1.0/2.0)Input p Touch screen, keyboard, keypadSensors GPU, GPU compass, accelerometer, camera (1.3-5mb+) l (1 3 5 b )Networking Bluetooth, Wifi, Bluetooth Wifi 3GScreen 320x240 up to 800x480
  • 53. HTC HD2Windows MobileFast CPU (1GHz)Big screen 4.3”, 800x480GPS, compass and accelerometerGood camera Depends on lighting conditionsHardware 3D Slow texture upload: slow video background rendering video-background
  • 54. HTC D DesireAndroidFast CPU (1GHz)Smaller screen 3.7”, 800x480Multi-touchHardware 3DGPS, compass andaccelerometer
  • 55. iPhone 4 PhApple iOS 4 0 4.0Faster CPU (1.2GHz)HighHi h screen resolution l i 3.5”, 960x640(Finally) camera APIMulti-touchHardware 3DGPS, compassGPS compass, accelerometerand gyroscope
  • 56. Hardware SensorsCamera (resolution, fps)C ( l i f ) Maker based/markerless tracking Video Vid overlap lGPS (resolution, update rate) Outdoor locationCompass Indoor/outdoor orientationAccelerometer Motion sensing, relative tilt sensing
  • 57. Programming Environments
  • 58. Mobile Development EnvironmentsNot like developing for desktops Wide range of different OS Limited CPU, low memory, poor graphics, no floating pointPopular Mobile OS Symbian (S bi C++, C bid S bi (Symbian C++ Carbide IDE) iPhone (Objective C) Android (Java, Native NDK wrapper) (J pp ) Windows Mobile (C#, C++, Visual Studio)Other Palm OS, Blackberry, Linux
  • 59. Programming Windows MobileVery similar to desktop Windows Almost identical API Unicode functions only U d f lDevelopment tools Embedded Visual C++ C# C++, - Deprecated, not suggested Visual Studio 2005 Visual Studio 2008 - Required for FPU usageForF overview of camera bugs look at i f b l k t
  • 60. Programming SymbianDevelopment tools Carbide.c++ Commercial version required for on-device debugging (important since emulator is bad…) SDK appropriate f your d for deviceMany quirks Crippled C i l d C++ support t Writing to static variables not allowed/recommended Cleanup Stack p API includes ~1500 classesMoving to Qt g
  • 61. Programming iPhoneHarsh restrictions from Apple Apps have to go through the apps storeXcodeX d IDE for development f d l Nice development toolsObjective-CObjective C Required for application development Can call into C/C++ code C/CCamera API support in iOS 4+ Can overlay on live video y
  • 62. AndroidHardware creators HTC, LG, Samsung, MotorolaWidelyWid l available phone il bl h Different form factors – tablets, phones, PC, etcMultiple versions and fragmentation Android 1.0, 1.6 Android 2.0, 2.1Free Tools Eclipse Development App Integrator
  • 63. Mobile Graphics
  • 64. Computer Graphics on Mobile Phones Small screen, limited input options Limited support for accelerated graphics Most phones have no GPU Mobile Graphics Libraries OpenGL ES ( , 2.0) p (1.0, ) - Cross platform, subset of OpenGL - C/C++ low level library Java M3G - Mobile 3D graphics API for J2ME platform - Object importer, scene graph library - Support from all major p pp j phone manufacturers
  • 65. OpenGL O GL ESSmall-footprintSmall footprint subset of OpenGL OpenGL is too large for embedded devices!Powerful, lP f l low-level API full functionality for 3D games l l API, f ll f i li f Can do almost everything OpenGL can (but only one way) Available A l bl on all key platforms ll k l f Software and hardware implementations availableFullyF ll extensible ibl Extensions like in OpenGLNo redundancy! Convenience functions removed
  • 66. OpenGL ES SLIDE 68 OpenGL ES vs. OpenGL (1.x) OpenGL OpenGL ES glBegin/glEnd 1 Primitive Types all no quads & polygons Data Types float, double int etc float double, int, etc… float, float fixed glDraw/Read Pixels glReadPixels only Textures T t 1D, 2D, 3D, cube 2D Stencil optional Window Bindings WGL, GLX, etc… EGL1: Except for Security Critical profile
  • 67. OpenGL ES SLIDE 69OpenGL ES on mobile devices Java Applications C++ Applications Scenegraph API S h Game G Middleware Middl M3G (JSR 184) Engine Engines
  • 68. OpenGL ES PerformanceFaster graphics (esp. hardware accelerated) ( )Longer batter performance (> 10%)
  • 69. VersionsTwo major tracks Not compatible, parallel rather than competitiveOpenGL ES 1 x 1.x Fixed function pipeline Suitable for software implementations All 1.x are backwards compatibleOpenGL ES 2 x 2.x Vertex and pixel shaders using GLSL ES All 2.x will be backwards compatible 2x
  • 70. Fixed Function (1 )F dF (1.x)
  • 71. Programmable (2.x)P bl (2 )
  • 72. OpenGL ES 1.x vs 2.0
  • 73. Tracking
  • 74. Mobile Augmented Reality’s goalCreate an affordable, massively multi-user, widespread platform © Tinmith, U. of South Australia
  • 75. Tracking is…Estimating the device‘s pose (position and orientation)Strictly in real time (30Hz) With high spatial precision (1cm, 1 degree) Robustly for operation by human user No unrealistic assumptions about HW Leaving enough power to other tasks (interaction graphics) (interaction,
  • 76. Tracking requirements for AR on phones Fast and efficient Form factor: light and robust Track simultaneously A large number of objects By a large number of users Requires little or no … q Device modification Manual calibration (targeting non-technical users) ( g g ) Instrumentation of the physical environment Low costs
  • 77. Tracking on mobile phonesVision-based tracking Marker-based tracking Model-based natural feature tracking Natural feature tracking in unknown environmentsSensor tracking GPS, inertial compass, gyroscope
  • 78. Tracking for Handheld AR SLIDE 80 Backpack-based 1. Höllerer et al. (1999), Piekarski & Thomas al. (2001), Reitmayr & Schmalstieg (2003) ( ), ( ), y g( )Laptop, HMDEnhanced GPS (DGPS / RTK) + inertial sensor for viewpoint trackingHand tracking w/ fiducial markers
  • 79. Tracking for Handheld AR SLIDE 81 Backpack-based 2. Kalkusch et al., 2002Video see-through HMD w/ camera gViewpoint Tracking w/ inside-out computer vision using markersARToolKit markers on walls installed and surveyed manually y y
  • 80. Tablet PC / UMPC-based 1. Schall et al., 2006Hybrid tracking on UMPC Camera fiducial marker trackingg When no marker in view inertial sensor + UWB tracking
  • 81. Tablet PC / UMPC-based 2. CAMERA LEDs Klein & Drummond, 2004Combining outside in (LED tracking for low accuracy robust pose) & outside-in accuracy,inside-out (edge-based tracking for high accuracy) computer vision
  • 82. PDA-based 1.BatPortal (Newman et al., 2001) ( , ) SHEEP (MacWilliams et al., 2003) PDA as thin client (rendering & Tracking by ART (external IR tracking on server + VNC) cameras + retroreflective target) Ultrasonic tracking Projection-based AR environ.
  • 83. PDA-based 2. Signpost on PDA (Wagner & Schmalstieg, 2003)First “truly” handheld AR platform: PDA + cameraStandalone, self-containedStandalone self contained AR systemOptimized fiducial marker tracking library
  • 84. History of non-AR Tracking on Phones (1)AR-PDA (Gausemeier et al., 2003) Kick Real (Paelke et al., 2004) Model-based Model based tracking Edge detection of real foot + collision PDA = thin client detection w/ virtual ball tracking off-loaded to server 2D tracking and limited interaction Not real-time (tailored to game)
  • 85. History of non-AR Tracking on Phones (2)PhoneGuide (Bruns et al., 2005) LightSense (Olwal, 2006) Neural network for recognizing visual External camera tracks cell phone LED features of museum artifacts Single user, only coarse position Combined w/ BT “tracker” tracking, no orientation Only object recognition O Border-case Border case of AR
  • 86. History of non-AR Tracking on Phones (3)Mosquito Hunt (Siemens, 2003)Marble Revolution (BitSide, 2004)Pingis (VTT,Pi i (VTT 2006) TinyMotion (Wang et al., 2006)Game control w/ optical GUI control & input on cell phonesflow techniques w/ image differencing & block correlation
  • 87. ARToolKit Tracking (Kato)ARToolKit - Computer vision based marker tracking libraries
  • 88. History of AR Tracking on Phones (1)2003 ARToolKit on PDA Wagner et at.2004 3D Marker on Phone Möhring et al. g2005 ARToolKit on Symbian Henrysson et al.
  • 89. Tracking for Handheld AR SLIDE 91Fiducial marker tracking on handheldsWagner et al., 2003 Möhring et al., 2004 Henrysson et al., 2006 Bucolo et al., 2005 Rohs, 2006
  • 90. History of AR Tracking on Phones (2) 2005 Visual Codes Rohs et at at. 2008 Advanced Marker Tracking Wagner et al. 2008 Natural Feature Tracking Wagner et al.
  • 91. What can we do on today‘s mobile phones? Typical specs 600+ MHz ~5MB of available RAM 160x120 - 320x240 at 15-30 Hz camera Possible to do Marker tracking in 5-15ms Natural feature tracking in 20-50ms
  • 92. Handheld AR Interfaces
  • 93. Handheld HCIConsider your userFollow good HCI principlesAdapt HCI guidelines for handheldsDesign to device constraintsRapid prototypingUser evaluation
  • 94. Consider Your User■ They are probably mobile able to use the interface with one hand■ They want quick access to application content content. Want enhanced interaction with the real world Interaction with the real world is the main focus■ They expect to be able to multitask start phone call, use another application, etc p pp
  • 95. Norman’s Principles of Good PracticeEnsure a high degree of visibility - allow the user to work out the current state of the system and the range of actions possible. f ti iblProvide feedback - continuous, clear information about the results of actions.Present a good conceptual model - allow the user to build up a picture of the way the system holds together, the relationships b h h l i hi between its different parts and h i diff d how to move from one state to the next.Offer good mappings g pp g - aim for clear, natural relationships between actions the user performs and the results they achieve.
  • 96. High L l D i Guidelines Hi h Level Design G id liFrom Shneiderman’s 8 desktop design g p g guidelines: Enable Frequent Users to Use Shortcuts Offer Informative Feedback Design Dialogs to Yield ClosureGong and Tarasewich’s guidelines:G dT i h’ id li Design for Small Devices Design for Limited and Split Attention Design for speed and recovery Allow for personalization Design for Enjoyment
  • 97. UI Device Constraints Comparing Desktop to Handheld Interfaces Screen Size Input Operation Multimedia ConnectivityDesktop > 1024 x 768 Mouse Two handed Millions of colours Wired Keyboard Stationary Graphics accel. Always On 5.1 AudioHandheld < 640 x 480 Stylus One handed 65K colours Wireless Touch Mobile No graphics accel. Maybe On Buttons Stereo audio
  • 98. Example: E m l O2 Active M A ti MenuHighly visualUse PDA buttons for inputLarge icons and easy to read textVisually indicate which tabs are scrollableApplication UI looks different from device UI
  • 99. iPhone GuidelinesMinimize required user input.Avoid unnecessary interactivity. y yProvide feedback when necessaryProvide fingertip sized target areas fingertip-sized areas.Avoid clutter and busy backgrounds.Express essential information succinctly. Make it obvious how to use your content. y
  • 100. iPhone Interface
  • 101. Designing for Device ConstraintsInput Device Touch, stylus, keyboard, buttons, keypad y y ypScreen Size, resolution ,Sensors Camera – frame rate image resolution rate, GPS – resolution, coverage Co pass accu acy Compass - accuracy
  • 102. Sample Handheld AR InterfacesCleanLarge Video ViewLarge IconsText Overlay
  • 103. Twitter ppiPhone applicationSee geo-located tweets in real worldTwitter.comTwitter com supports geo tagging
  • 104. Wikitude – Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Bl hh Blah Bl Blah Bl h Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah
  • 105. Information Filtering
  • 106. Information FilteringInformation Filtering (Julier et al. ’00) al• Remove clutter by goal- and distance based filtering• User’s task is route finding: Sniper and relevant buildings are displayed;objects, which are determined to be unnecessary, removed bj t hi h d t i dt b d
  • 107. HMD vs Handheld AR Interface Wearable AR W bl HandHeld AR Output: Display Input & Output Input
  • 108. Handheld Interface MetaphorsTangible AR Lens Viewing Look through screen into AR scene Interact with screen to interact with AR content - Eg Invisible TrainTangible AR Lens Manipulation Select AR object and attach to device Use the motion of the device as input - Eg AR Lego
  • 109. Handheld Display vs Fixed Display Experiment comparing handheld moving, to handheld button input, small fixed display, desktop display, large plasma Users performed (1) navigation task, (2) selection task Moving handheld display provided greater perceived FOV, higher degree of presence, faster completion timeJ. Hwang, J. Jung, G. Kim. Hand-held Virtual Reality: A Feasibility Study. In proceedings of VRST 2006
  • 110. Search Task Completion Time
  • 111. FOV, Presence and Immersion Perceived FOV and Actual FOV (deg. marked by subjects)70 64 58 6060 5250 4540 30 33 30 31 30 Perceived FOV30 Actual FOV2010 0 7 Motion Button Small 17 screen 42 screen 5.7 6 5.4 based hh based hh screen 5 4.7 4.7 4.9 4.3 4.5 4.3 4.3 4.1 4 Presence 3 Immersion 2 1 0 Motion Button Small 17 screen 42 screen based hh based hh screen
  • 112. Rapid PrototypingSpeed development time by using quick hardware mockups p p y gq p handheld device connected to PC LCD screen USB phone keypad CameraCan use PC development tools Flash, Visual Basic, etc
  • 113. Mobile Physical PrototypingBug Labs source hardware modules, eachpproducing one or more services. gModules snap together physically and theservices connect together logically to i h l i llenable users to easily build applications.
  • 114. Software PrototypingPython Symbian (HIT Lab NZ) stbTracker wrapper Access to SMS, Bluetooth, GPS Rapid development
  • 115. import e32 import appuifw from gles import *if e32.s60_version_info>=(3,0): import imp magnet=imp.load_dynamic(Magnet, t i l d d i (M t c:sysbinMagnet.pyd) bi M t d)else: import Magnetfrom Magnet import#Define Modeldef frameback(num_markers): if (num markers > -1): (num_markers 1): glMatrixMode(GL_PROJECTION)#Draw graphics… = landscape‘ # Use full frameSetCameraCallback(frameback) # Register callbackcreateCamera() # Define cameraInitGLES() # Start Open GLTrackerInit() # Start trackerInitCamera() # Start camera
  • 116. Design GuidelinesApply handheld HCI guidelines for on screen content on-screen - large buttons, little text input, etcDesign physical + virtual interface elementsPick appropriate interface metaphor pp p p - “handheld lens” approach using handheld motion - Tangible AR for AR overlayBuild prototypesContinuously evaluate application
  • 117. AR Browsers
  • 118. AR BrowsersCommercial outdoor AR applications Junaio, Layar, Wikitude, etcAll have their own language specifications Wikitude – ARML Junaio - XMLNeed for common standard Based on existing standards for geo-located content etc Support for dynamic/interactive content Easier to author mobile AR applications Easy to render on AR browsers
  • 119. Layar
  • 120. Junaio
  • 121. Hello World Exampleecho ""<?xml version="1.0" encoding="UTF-8"?> " " " "<results><poi id="1" interactionfeedback="none"> <name><![CDATA[Hello World POI]]></name> <description><![CDATA[[This is my first POI.]]></description> <l>48.1385,11.5750,0</l> , , <o>0,0,0</o><mime-type>text/plain</mime-type><thumbnail></<thumbnail>http://www junaio com/publisherDownload/tutorial/icon jpg</thumbnail><icon></icon></poi></results>"
  • 122. KHARMA + Argon: A KML/HTML Architecture and Browser for AR Applications and Games Blair MacIntyre and Al Hill Bl i M I d AlexSchool of Interactive Computing, Georgia Institute of Technology 145
  • 123. KHARMA:KHARMA KML/HTML Augmented Reality Mobile Architecture research tools { media hackers our past focus { skilled computationalists savvy technical designers Ygrasil IDE current focus { general public breadth of adoption Unity AR ToolkitDesigners AR Toolkit KHARMA 146
  • 124. KHARMA: KML/HTML Augmented Reality Mobile Architecture Problem: limited authoring tools for mobile AR limited expression (coord, name, desc, link) vs higher hurdle of 3D no accepted standard and proprietary client protocol limited client-side interactivity is more akin to Web 1 0 1.0 Solution: HTML with KML combines What? and How? with Where? • allows extensive client side (albeit 2D) interactivity and expressivity • two the most broadly adopted standards for presentation and geo location • HTTP server distribution, CSS and Javascript allow for true Web 2.0 content + KML HTML 147
  • 125. KHARMAArchitecture with four components Channel servers - delivering individual channels of AR content, Tracking servers - providing content related to location Tracking, Tracking infrastructure servers - delivering information about the physical environment, Mobile client - for generating the resulting augmentations
  • 126. KML already supports HTML• description tag accepts CDATA enclosed markup - but no global styling of scripting support• no control over balloon styling• no control over balloon position and orientation• no relative positioning <Placemark id="culc_center"> <name>CULC Visualization</name> <description><![CDATA[ <div id="culc_image">Georgia T h> <di id " l i ">G i Tech> <img src=""></div> </description> <Balloon> <location> <latitude>34.0</latitude> <longitude>--84.5</longitude> </location> <orientation> <heading>31</heading> </orientation> </Balloon> </Placemark>
  • 127. KARML extension to KML <Placemark id="culc_center"> <name>CULC Visualization</name>• added undecorated displayMode <description><![CDATA[ <div id="culc_image"><img src=""></div> </description> <Style>• added Balloon element <BalloonStyle> <displayMode>undecorated</displayMode> - similar in nature to Model element </BalloonStyle> </Style> <Balloon> <locationMode targetId=”culc_g g geospot”>relative</locationMode> p• relative locationMode <location> - accepts “units” attribute <latitude units="meters">4.0</latitude> <longitude units="meters">-2.5</longitude> </location> <orientation> <heading>31</heading> </orientation> </Balloon> </Placemark> 151
  • 128. GPS Located Content <Placemark>Geospot <name>GeoSpot</name> <description>a surveyed GeoSpot </description> Surveyed location <Camera> <!-- camera element for GeoSpot --> p Moves camera to <longitude>-84.394135</longitude> <!- GPS coordinates --> <latitude>33.76083</latitude> fixed location <altitude>0</altitude> <TimeStamp> <! when GeoSpot was surveyed --> <!-- > <when>1997-07-16T10:30:15+03:00</when> </TimeStamp> <Icon> <href></href> </Icon> </Camera> <Point> <! location displayed on map or within browser view --> <!-- > <coordinates>-84.394135,33.76083</coordinates> </Point> </Placemark>
  • 129. For more information, please visit: http://www argon gatech edu/
  • 130. Developing AR Experiences Sony CSL © 2004
  • 131. Game Case Study
  • 132. Resources
  • 133. ResourcesBooksMobile Interaction Design Matt J M Jones and Gary Marsden dG M dDesigning for Small Screens Studio 7.5Handheld Usability Scott WeissDesigning the Mobile User Experience Barbara Ballard
  • 134. Developer Guidelines Palm Zen of Palm guidelines Motorola iPhone Human Interface Guidelines
  • 135. Handheld HCI Design WebsitesDo’s and Don’ts of PocketPC design bili special i i l interest group – h dh ld usability handheld bili Mobile website Coders Website mobilecoders com/Articles/mc-01 aspUniv of Waikato Handheld Group Interaction Website
  • 136. Platform – Recommended readingLots of low level informationon the complete ARM family p yValuable tool for driver andframework developersNot that important for pureapplication developers
  • 137. Platform – Recommended readingVery low level and targeted for PCsMost information outdated on PCEffective memory usage oneof the most importantoptimization strategieson mobile devices!
  • 138. Tracking – Recommended readingLots of the basics on theComputer Vision you will p yneed for AR trackingSeveral code and pseudo-code pseudo codesnippets
  • 139. Tracking – Recommended readingAll about the geometryyou will need fora tracking system Camera models Projection Epipolar geometry Homographies H hi …
  • 140. Graphics – Recommended reading Mobile 3D Graphics (all b t O GL ( ll about OpenGL ES 1.x) 1 ) OpenGL ES 2.0 Programming G Guide OpenGL ES 2.0 Man Pages ShaderX7 Chapter on “ Augmented Reality on Mobile Phones”
  • 141. OpenGL ES ResourcesKhronos Group OpenGL ES Page ES 2.0 Book OpenGL ES 2.0 Emulator