Lecture 5: Mobile AR       Mark Billinghurst mark.billinghurst@hitlabnz.org    k billi h t@hitl b           HIT Lab NZ   U...
AR on mobile phonesLow cost, widely spread platform  Billions of phones deployed  People know how to use them  Strong dema...
Why would you use phones?Robust and fool-proof  People know how to use their mobile phones…Variety of supported devicesSel...
Why would you not use phones?Compared to PC-based setups  Less processing power and memory       p        gp              ...
Other limitations of handheld ARUsual limitations in mobile HCI  Small screen   -   Less information possible to display  ...
Mobile AR History
Evolution of Mobile AR                E l        fM bl            Camera phone                                            ...
Handheld DisplaysTethered Applications    Fitzmaurice Chameleon (1994)                           (    )    Rekimoto’s Tran...
Handheld AR Display - Tethered1995, 1996 Handheld AR   ARPad,   ARPad Cameleon   Rekimoto’s NaviCam, Transvision   Tethere...
Mobile AR: Touring Machine (1997)Columbia University  Feiner, MacIntyre, Höllerer, WebsterCombines  See through head mount...
MARS ViewVirtual tags overlaid on th real worldVi t l t         l id    the   l    ld“Information in place”
Backpack/Wearable AR1997 Backpack AR   Feiner’s Touring Machine   AR Quake (Thomas)   Tinmith (Piekarski)   MCAR (Reitmayr...
Mobile AR - Hardware                    RTK correction Antenna    GPS                                             Example ...
Sharp J-SH041997 Philip Kahn invents camera phone1999 First commercial camera phone
Millions of Camera Phones12001000 800                                                      DSC 600                        ...
Handheld AR – Thin Client2001 BatPortal (AT&T Cambridge)     PDA used as I/O device     Wireless connection to workstation...
Mobile Phone AR – Thin Client2003 ARphone (Univ. of Sydney)    Transfer images via Bluetooth (slow – 30 sec/image)        ...
Early Phone Computer Vision Apps2003 – Mozzies Game - Best mobile gameOptical motion flow detecting phone orientationSieme...
Computer Vision on Mobile PhoneCameras and Phone CPU sufficient for computer visionapplicationsPattern Recognition (Static...
Handheld AR – Self Contained2003 PDA-based AR     ARToolKit port to PDA     Studierstube ported to PDA     AR Kanji Educat...
Mobile Phone AR – Self Contained2004 M bil Phone AR     Mobile Ph    Moehring, Bimber    Henrysson (ARToolKit)    Camera, ...
Location Aware PhonesMotorola Droid        Nokia Navigator
Real World Information OverlayTag real world locations  GPS + Compass input  Overlay graphics data on live videoApplicatio...
Layar – www.layar.com
HIT Lab NZ Android AR PlatformArchitectural ApplicationLoads 3D models  a OBJ/MTL formatPositions content in space        ...
HIT Lab NZ Mobile Outdoor AR
History of Handheld and Mobile AR1995 Handheld Display: NaviCam, AR-PAD, Transvision1997 Wearable AR: Touring Machine, AR ...
1996        Mobile AR by Weight                        2003                                                    2007       ...
2011 S                     State of the Art                            f h AHandheld Hardware available   PDA, mobile phon...
Mobile AR CompaniesMobile AR  GPS + compassMany CompaniesM    C     i  Layar  Wikitude  Acrossair  PressLite  Yelp  Robot ...
$2 million USD in 2010$732 million USD in 2014
Handset ManufacturersQualcomm  $100 million USD investmentNokiaN ki  25+ people in NRCSamsung  Exploring the spaceApple  5...
QualcommAcquired Imagination   q          gOctober 2010 - Releases free Android AR SDKComputer vision tracking - marker, m...
Rock-em Sock-emSharedSh d AR D  DemoMarkerless tracking
AppleiPhone 4 SDK supports direct camera access               ppLaunches AR theme on App Store
> 500 AR apps on App Store
Developing AR applications
Mobile AR TechnologyInvolves  Tracking         g  Content Loading  Rendering/3D graphics           g     g p  User Interfa...
Scientific challengesAR requires (unlike related disciplines)  Strict real time operation (30Hz)   -   Unlike Ubicomp or m...
How does a basic AR application work?Main loop Get a video frame from the camera Estimate the position and the orientation...
Studierstube ES FrameworkTypical AR application                                  ApplicationsframeworkDeveloped at TU     ...
End UserApplication ppProgrammingLibrariesLib iOS/Low Level APIHardware
The Studierstube ES frameworkTh S d        b     f       k                                Applications                    ...
Platform
What are the challenges?Experiences with embedded platforms requiredMany platforms (operating systems)M      l f     (    ...
Processing power of mobile phonesWeak processing power  ~1GHz,  ~1GHz Single core  Often no floating point unit   -   Floa...
So what are the common problems?Bad camera quality under low lighting  Noise, motion blur       ,  Strongly varies with di...
Platforms for handheld AR               Pl f      f h dh ld                           Pros                                ...
Worldwide smartphone market share   1Q09            1Q10                                        Symbian                   ...
2011 US Market Share
What makes a device interesting for AR?  Open and easy to program  Good camera  Fast CPU, FPU is a plus  Good H/W 3D suppo...
Typical Smart Phone HardwareCPU   300-800+ MhzGPU   None, or Power VR Chip (OpenGL ES1.0/2.0)Input  p   Touch screen, keyb...
HTC HD2Windows MobileFast CPU (1GHz)Big screen  4.3”, 800x480GPS, compass and accelerometerGood camera  Depends on lightin...
HTC D                      DesireAndroidFast CPU (1GHz)Smaller screen  3.7”, 800x480Multi-touchHardware 3DGPS, compass and...
iPhone 4                   PhApple iOS 4 0          4.0Faster CPU (1.2GHz)HighHi h screen resolution                l i  3...
Hardware SensorsCamera (resolution, fps)C      (    l i     f )  Maker based/markerless tracking  Video  Vid overlap      ...
Programming Environments
Mobile Development EnvironmentsNot like developing for desktops  Wide range of different OS  Limited CPU, low memory, poor...
Programming Windows MobileVery similar to desktop Windows   Almost identical API   Unicode functions only   U    d f      ...
Programming SymbianDevelopment tools  Carbide.c++  Commercial version required for on-device debugging (important  since e...
Programming iPhoneHarsh restrictions from Apple   Apps have to go through the apps storeXcodeX d IDE for development      ...
AndroidHardware creators  HTC, LG, Samsung, MotorolaWidelyWid l available phone         il bl h  Different form factors – ...
Mobile Graphics
Computer Graphics on Mobile Phones Small screen, limited input options Limited support for accelerated graphics    Most ph...
OpenGL                  O GL ESSmall-footprintSmall footprint subset of OpenGL  OpenGL is too large for embedded devices!P...
OpenGL ES                     SLIDE 68            OpenGL ES vs. OpenGL (1.x)                                            Op...
OpenGL ES     SLIDE 69OpenGL ES on mobile devices          Java Applications              C++ Applications                ...
OpenGL ES PerformanceFaster graphics (esp. hardware accelerated)                (                         )Longer batter p...
VersionsTwo major tracks  Not compatible, parallel rather than competitiveOpenGL ES 1 x          1.x  Fixed function pipel...
Fixed Function (1 )F dF           (1.x)              http://www.khronos.org/opengles/2_X/
Programmable (2.x)P        bl (2 )            http://www.khronos.org/opengles/2_X/
OpenGL ES 1.x vs 2.0
Tracking
Mobile Augmented Reality’s goalCreate an affordable, massively multi-user, widespread platform      © Tinmith, U. of South...
Tracking is…Estimating the device‘s pose (position and orientation)Strictly in real time (30Hz)  With high spatial precisi...
Tracking requirements for AR on phones  Fast and efficient  Form factor: light and robust  Track simultaneously    A large...
Tracking on mobile phonesVision-based tracking  Marker-based tracking  Model-based natural feature tracking  Natural featu...
Tracking for Handheld AR                    SLIDE 80                         Backpack-based 1. Höllerer et al. (1999), Pie...
Tracking for Handheld AR                SLIDE 81                     Backpack-based 2.                  Kalkusch et al., 2...
Tablet PC / UMPC-based 1.                 Schall et al., 2006Hybrid tracking on UMPC  Camera   fiducial marker trackingg  ...
Tablet PC / UMPC-based 2.                                                   CAMERA                                  LEDs  ...
PDA-based 1.BatPortal (Newman et al., 2001)          (             ,     )     SHEEP (MacWilliams et al., 2003)  PDA as th...
PDA-based 2.   Signpost on PDA (Wagner & Schmalstieg, 2003)First “truly” handheld AR platform: PDA + cameraStandalone, sel...
History of non-AR Tracking on Phones (1)AR-PDA (Gausemeier et al., 2003)     Kick Real (Paelke et al., 2004)  Model-based ...
History of non-AR Tracking on Phones (2)PhoneGuide (Bruns et al., 2005)             LightSense (Olwal, 2006) Neural networ...
History of non-AR Tracking on Phones (3)Mosquito Hunt (Siemens, 2003)Marble Revolution (BitSide, 2004)Pingis (VTT,Pi i (VT...
ARToolKit Tracking (Kato)ARToolKit - Computer vision based marker tracking librarieshttp://artoolkit.sourceforge.net/
History of AR Tracking on Phones (1)2003   ARToolKit on PDA   Wagner et at.2004   3D Marker on Phone   Möhring et al.     ...
Tracking for Handheld AR                 SLIDE 91Fiducial marker tracking on handheldsWagner et al., 2003         Möhring ...
History of AR Tracking on Phones (2) 2005    Visual Codes    Rohs et at            at. 2008    Advanced Marker Tracking   ...
What can we do on today‘s mobile phones? Typical specs    600+ MHz    ~5MB of available RAM    160x120 - 320x240 at 15-30 ...
Handheld AR Interfaces
Handheld HCIConsider your userFollow good HCI principlesAdapt HCI guidelines for handheldsDesign to device constraintsRapi...
Consider Your User■ They are probably mobile     able to use the interface with one hand■ They want quick access to applic...
Norman’s Principles of Good PracticeEnsure a high degree of visibility - allow the user to work out the current state of t...
High L l D i Guidelines       Hi h Level Design G id liFrom Shneiderman’s 8 desktop design g                              ...
UI Device Constraints  Comparing Desktop to Handheld Interfaces           Screen Size    Input      Operation    Multimedi...
Example:     E m l O2 Active M              A ti MenuHighly visualUse PDA buttons for inputLarge icons and easy to read te...
iPhone GuidelinesMinimize required user input.Avoid unnecessary interactivity.                  y            yProvide feed...
iPhone Interface
Designing for Device ConstraintsInput Device  Touch, stylus, keyboard, buttons, keypad           y       y                ...
Sample Handheld AR InterfacesCleanLarge Video ViewLarge IconsText Overlay
Twitter 360www.twitter-360.com        ppiPhone applicationSee geo-located tweets in real worldTwitter.comTwitter com suppo...
Wikitude – www.mobilizy.com              Blah                     Blah Blah          Blah    Blah                         ...
Information Filtering
Information FilteringInformation Filtering (Julier et al. ’00)                                 al• Remove clutter by goal-...
HMD vs Handheld AR Interface  Wearable AR  W    bl                  HandHeld AR                Output:                Disp...
Handheld Interface MetaphorsTangible AR Lens Viewing   Look through screen into AR scene   Interact with screen to interac...
Handheld Display vs Fixed Display    Experiment comparing handheld moving, to handheld button input, small    fixed displa...
Search Task Completion Time
FOV, Presence and Immersion           Perceived FOV and Actual FOV (deg. marked by subjects)70                            ...
Rapid PrototypingSpeed development time by using quick hardware mockups p          p           y     gq                   ...
Mobile Physical PrototypingBug Labshttp://www.buglabs.net/Open source hardware modules, eachpproducing one or more service...
Software PrototypingPython Symbian (HIT Lab NZ)  stbTracker wrapper  Access to SMS, Bluetooth, GPS  Rapid development
import e32    import appuifw    from gles import *if e32.s60_version_info>=(3,0):    import imp    magnet=imp.load_dynamic...
Design GuidelinesApply handheld HCI guidelines for on screen content                                       on-screen   - l...
AR Browsers
AR BrowsersCommercial outdoor AR applications  Junaio, Layar, Wikitude, etcAll have their own language specifications  Wik...
Layar
Junaio
Hello World Exampleecho ""<?xml version="1.0" encoding="UTF-8"?>                       "    "          "      "<results><p...
KHARMA + Argon: A KML/HTML Architecture and Browser for  AR Applications and Games Blair MacIntyre and Al Hill Bl i M I   ...
KHARMA:KHARMA KML/HTML Augmented Reality Mobile Architecture                       research tools     {     media hackers ...
KHARMA: KML/HTML Augmented Reality Mobile Architecture Problem: limited authoring tools for mobile AR       limited expres...
KHARMAArchitecture with four components Channel servers  - delivering individual channels of AR content, Tracking servers ...
KML already supports HTML• description tag accepts CDATA enclosed markup  - but no global styling of scripting support• no...
KARML extension to KML                                  <Placemark id="culc_center">                                      ...
GPS Located Content                     <Placemark>Geospot             <name>GeoSpot</name>                    <descriptio...
For more information, please visit:    http://www.argon.gatech.edu/    http://www argon gatech edu/
Developing AR Experiences                            Sony CSL © 2004
Game Case Study
Resources
ResourcesBooksMobile Interaction Design    Matt J    M Jones and Gary Marsden                  dG     M dDesigning for Sma...
Developer Guidelines    Palmhttp://www.access-company.com/developers/documents/docs/ui/UI_Design.html    Zen of Palm guide...
Handheld HCI Design WebsitesDo’s and Don’ts of PocketPC designhttp://www.pocketpcmag.com/_archives/Nov04/Commandements.asp...
Platform – Recommended readingLots of low level informationon the complete ARM family            p               yValuable...
Platform – Recommended readingVery low level and targeted for PCsMost information outdated on PCEffective memory usage one...
Tracking – Recommended readingLots of the basics on theComputer Vision you will     p             yneed for AR trackingSev...
Tracking – Recommended readingAll about the geometryyou will need fora tracking system  Camera models  Projection  Epipola...
Graphics – Recommended reading Mobile 3D Graphics (all b t O GL ( ll about OpenGL ES 1.x)                      1 ) OpenGL ...
OpenGL ES ResourcesKhronos Group OpenGL ES Page  http://www.khronos.org/opengles/OpenGL ES 2.0 Book  http://www.opengles-b...
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
COSC 426 Lect. 5 - Mobile AR
Upcoming SlideShare
Loading in...5
×

COSC 426 Lect. 5 - Mobile AR

2,949

Published on

Lecture 5 of the COSC 426 graduate course on Augmented Reality. This lecture provides an overview of Mobile Augmented Reality. The Lecture is given by Mark Billinghurst of the HIT Lab NZ at the University of Canterbury

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,949
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
111
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

COSC 426 Lect. 5 - Mobile AR

  1. 1. Lecture 5: Mobile AR Mark Billinghurst mark.billinghurst@hitlabnz.org k billi h t@hitl b HIT Lab NZ University of Canterbury
  2. 2. AR on mobile phonesLow cost, widely spread platform Billions of phones deployed People know how to use them Strong demand from commercial side Huge chance for AR!Target practical applications Easy to use Quality graphics Robust tracking 15-30 Hz overall frame rate
  3. 3. Why would you use phones?Robust and fool-proof People know how to use their mobile phones…Variety of supported devicesSelf containedSelf-contained operation Enough processing power to do everything nativelyUltra-mobileUltra mobileLow cost devices Even better: people already own the target hardware! !A very unique chance for bringing AR to the masses
  4. 4. Why would you not use phones?Compared to PC-based setups Less processing power and memory p gp y Harder to program, debug and deployHardware difficult or impossible to extendSmall number of available librariesTypically l littleT i ll only li l experience i research groups i in hSo many different devices and operating systems
  5. 5. Other limitations of handheld ARUsual limitations in mobile HCI Small screen - Less information possible to display - Less immersion Limited input …Other limitations You see through the camera and not through the phone Switch attention between background and phone Strain factor of holding the phone up Social issues with pointing the phone at people p g p p p
  6. 6. Mobile AR History
  7. 7. Evolution of Mobile AR E l fM bl Camera phone Camera phone - Thin client ARWearable Wearable ARComputers Camera phone - Self contained AR PDAs Handheld -Thin client AR AR Displays p y PDAs -Self contained AR 1995 1997 2001 2003 2004
  8. 8. Handheld DisplaysTethered Applications Fitzmaurice Chameleon (1994) ( ) Rekimoto’s Transvision (1995) Tethered LCD PC Processing and Tracking
  9. 9. Handheld AR Display - Tethered1995, 1996 Handheld AR ARPad, ARPad Cameleon Rekimoto’s NaviCam, Transvision Tethered LCD PC Processing and Tracking
  10. 10. Mobile AR: Touring Machine (1997)Columbia University Feiner, MacIntyre, Höllerer, WebsterCombines See through head mounted display GPS t ki tracking Orientation sensor Backpack PC ( p (custom) ) Tablet input
  11. 11. MARS ViewVirtual tags overlaid on th real worldVi t l t l id the l ld“Information in place”
  12. 12. Backpack/Wearable AR1997 Backpack AR Feiner’s Touring Machine AR Quake (Thomas) Tinmith (Piekarski) MCAR (Reitmayr) Bulky, HMD based
  13. 13. Mobile AR - Hardware RTK correction Antenna GPS Example self-built working Antenna solution with PCI-based 3D graphics PCI based HMD Controller PCI 3D Graphics Board Tracker Controller PC104 Sound Card DC to DC Wearable Converter CPU Computer PC104 PCMCIA Battery GPS RTK Hard Drive correction Radio Serial PortsColumbia Touring Machine
  14. 14. Sharp J-SH041997 Philip Kahn invents camera phone1999 First commercial camera phone
  15. 15. Millions of Camera Phones12001000 800 DSC 600 Phone 400 200 0 2002 2003 2004 2005 2006 2007 2008 2009 2010
  16. 16. Handheld AR – Thin Client2001 BatPortal (AT&T Cambridge) PDA used as I/O device Wireless connection to workstation Wi l i k i Room-scale ultrasonic tracking (Bat)2001 AR-PDA (C Lab) PDA thin graphics client Remote image processing www.ar-pda.com
  17. 17. Mobile Phone AR – Thin Client2003 ARphone (Univ. of Sydney) Transfer images via Bluetooth (slow – 30 sec/image) g ( g ) Remote processing – AR Server
  18. 18. Early Phone Computer Vision Apps2003 – Mozzies Game - Best mobile gameOptical motion flow detecting phone orientationSiemens SX1 – S bi 120Mh VGA CameraSi Symbian, 120Mhz, C2005 – Marble Revolution (Bit-Side GmbH)Winner of Nokias Series 60 Challenge 2005Wi f N ki S i Ch ll2005 – SymBall (VTT)
  19. 19. Computer Vision on Mobile PhoneCameras and Phone CPU sufficient for computer visionapplicationsPattern Recognition (Static Processing) g ( g) QR Code Shotcode (http://www.shotcode.com/)Motion Flow (2D IM i Fl Image P Processing) i ) GestureTek - http://www.gesturetekmobile.com/ TinyMotion3D Pose Calculation Augmented Reality
  20. 20. Handheld AR – Self Contained2003 PDA-based AR ARToolKit port to PDA Studierstube ported to PDA AR Kanji Educational App. Mr Virtuoso AR character Wagner’s Invisible Train - Collaborative AR
  21. 21. Mobile Phone AR – Self Contained2004 M bil Phone AR Mobile Ph Moehring, Bimber Henrysson (ARToolKit) Camera, processor, display together
  22. 22. Location Aware PhonesMotorola Droid Nokia Navigator
  23. 23. Real World Information OverlayTag real world locations GPS + Compass input Overlay graphics data on live videoApplications Travel guide, Advertising, etcEg: Mobilizy Wikitude (www.mobilizy.com) Android based, Public API releasedOther companies Layar, AcrossAir, Tochnidot, RobotVision, etc
  24. 24. Layar – www.layar.com
  25. 25. HIT Lab NZ Android AR PlatformArchitectural ApplicationLoads 3D models a OBJ/MTL formatPositions content in space p GPS, compassIntuitive user interface toolkit to modify the modelConnects to back end model database
  26. 26. HIT Lab NZ Mobile Outdoor AR
  27. 27. History of Handheld and Mobile AR1995 Handheld Display: NaviCam, AR-PAD, Transvision1997 Wearable AR: Touring Machine, AR Quake g2001 Handheld AR – Thin Client: AR-PDA, Bat Portal2003 Handheld AR – Self contained: Invisible Train2003 Mobile Phone – 2D Vision: Mozzies, Symball2003 Mobile Phone – Thin Client: ARphone2004 Mobile Phone – Self contained: Moehring, Symbian
  28. 28. 1996 Mobile AR by Weight 2003 2007 Scale it down: Scale it down more: Vesp R Vesp‘R [Kruijff ISMAR07]: Smartphone…$500 S t h $500 …Sony UMPC 1.1GHz …All-in-oneBackpack+HMD: …1.5kg …0.1kg…5-8kg 5 8k …still >$5K till …billions of units
  29. 29. 2011 S State of the Art f h AHandheld Hardware available PDA, mobile phones, external cameras Sensors: GPS, accelerometer, compassSoftware Tools are Available Tracking: ARToolKitPlus, QCAR Graphics: OpenGL ES Authoring: Studierstube, Layar, Wikitude, UnifyeWhat is needed: High level authoring tools Content development tools Novel interaction techniques User evaluation and usability
  30. 30. Mobile AR CompaniesMobile AR GPS + compassMany CompaniesM C i Layar Wikitude Acrossair PressLite Yelp Robot vision Etc.. E
  31. 31. $2 million USD in 2010$732 million USD in 2014
  32. 32. Handset ManufacturersQualcomm $100 million USD investmentNokiaN ki 25+ people in NRCSamsung Exploring the spaceApple 586 AR Applications on App StoreGoogle Google goggles/Android AR Applications g g gg pp
  33. 33. QualcommAcquired Imagination q gOctober 2010 - Releases free Android AR SDKComputer vision tracking - marker, markerless p gIntegrated with Unity 3D rendererhttp://developer.qualcomm.com/ar p p q
  34. 34. Rock-em Sock-emSharedSh d AR D DemoMarkerless tracking
  35. 35. AppleiPhone 4 SDK supports direct camera access ppLaunches AR theme on App Store
  36. 36. > 500 AR apps on App Store
  37. 37. Developing AR applications
  38. 38. Mobile AR TechnologyInvolves Tracking g Content Loading Rendering/3D graphics g g p User Interface Application Design Evaluation
  39. 39. Scientific challengesAR requires (unlike related disciplines) Strict real time operation (30Hz) - Unlike Ubicomp or mobile information systems p y High spatial precision (1cm, 1 degree) - Unlike location-based services Robustness for operation by human user - Unlike many computer vision methods in automation etc.Mobile hM bil phone AR requires (in addition) i (i ddi i ) No thin client! Same l l of performance as desktop AR S level f f d k - New algorithms must be orders of magnitude more efficient No unrealistic assumptions about HW
  40. 40. How does a basic AR application work?Main loop Get a video frame from the camera Estimate the position and the orientation - computer vision, sensor input (GPS, compass) p p ( p ) Render the augmented scene (video + virtual content) ) Render GUI Process user input Update application status
  41. 41. Studierstube ES FrameworkTypical AR application ApplicationsframeworkDeveloped at TU StudierstGraz Platformtube Software Stack a
  42. 42. End UserApplication ppProgrammingLibrariesLib iOS/Low Level APIHardware
  43. 43. The Studierstube ES frameworkTh S d b f k Applications o User Interface - Application Studie Content erstube Software S Graphics Stack Tracking Platform Platform
  44. 44. Platform
  45. 45. What are the challenges?Experiences with embedded platforms requiredMany platforms (operating systems)M l f ( i )Slow CPUs (low clock rates, often no FPU) Difficulties with tracking Difficulties with visualizations that require a lot of data processing iSlow memory accessNo or weak hardware 3D acceleration Difficulties with graphics
  46. 46. Processing power of mobile phonesWeak processing power ~1GHz, ~1GHz Single core Often no floating point unit - Floating po t co e ~40x s owe t a integer co e oat g point code 0 slower than tege code - Fixed point problematic for many algorithms - Requires good math libraryCode optimized for phones runs 5-10x slower on ahigh-end phone than on an average PCNot going to change dramatically due to battery power
  47. 47. So what are the common problems?Bad camera quality under low lighting Noise, motion blur , Strongly varies with different phonesSmall memory Keeping large databases in memory is problematicSlow memory Low clock rate - Processing large memory areas is prohibitive g g y p - Typical CV building blocks (SVD, image processing) are too demanding Slow data transfers between CPU and GPU
  48. 48. Platforms for handheld AR Pl f f h dh ld Pros ConsWindows  Easy to program and debug Camera drivers are not always goodMobile Lots of devices Largest installed basis Hard to program and debug, SDK changes Symbian Good devices often, usually slow CPUs Very nice hardware Very nice hardware Camera API only with Camera API only with OS4iPhone Hype factor Objective C as main language Increasing number of devicesAndroid Java as main language Java as main language Hype factorLinux (LiMo,  Full Linux support, limited Large set of different libraries Large set of different librariesMaemo)M ) handsets  for AR h d f ARRIM  Widely spread in the US Java onlyBlackberryPalm  Only very few devices so far Nice hardwareWebOS No native SDK so far
  49. 49. Worldwide smartphone market share 1Q09 1Q10 Symbian RIM iPhone Android Source: Gartner http://www.gartner.com
  50. 50. 2011 US Market Share
  51. 51. What makes a device interesting for AR? Open and easy to program Good camera Fast CPU, FPU is a plus Good H/W 3D support Large installed basis g Easy access to devices GPS, accelerometer, compass Enough memory/storage
  52. 52. Typical Smart Phone HardwareCPU 300-800+ MhzGPU None, or Power VR Chip (OpenGL ES1.0/2.0)Input p Touch screen, keyboard, keypadSensors GPU, GPU compass, accelerometer, camera (1.3-5mb+) l (1 3 5 b )Networking Bluetooth, Wifi, Bluetooth Wifi 3GScreen 320x240 up to 800x480
  53. 53. HTC HD2Windows MobileFast CPU (1GHz)Big screen 4.3”, 800x480GPS, compass and accelerometerGood camera Depends on lighting conditionsHardware 3D Slow texture upload: slow video background rendering video-background
  54. 54. HTC D DesireAndroidFast CPU (1GHz)Smaller screen 3.7”, 800x480Multi-touchHardware 3DGPS, compass andaccelerometer
  55. 55. iPhone 4 PhApple iOS 4 0 4.0Faster CPU (1.2GHz)HighHi h screen resolution l i 3.5”, 960x640(Finally) camera APIMulti-touchHardware 3DGPS, compassGPS compass, accelerometerand gyroscope
  56. 56. Hardware SensorsCamera (resolution, fps)C ( l i f ) Maker based/markerless tracking Video Vid overlap lGPS (resolution, update rate) Outdoor locationCompass Indoor/outdoor orientationAccelerometer Motion sensing, relative tilt sensing
  57. 57. Programming Environments
  58. 58. Mobile Development EnvironmentsNot like developing for desktops Wide range of different OS Limited CPU, low memory, poor graphics, no floating pointPopular Mobile OS Symbian (S bi C++, C bid S bi (Symbian C++ Carbide IDE) iPhone (Objective C) Android (Java, Native NDK wrapper) (J pp ) Windows Mobile (C#, C++, Visual Studio)Other Palm OS, Blackberry, Linux
  59. 59. Programming Windows MobileVery similar to desktop Windows Almost identical API Unicode functions only U d f lDevelopment tools Embedded Visual C++ C# C++, - Deprecated, not suggested Visual Studio 2005 Visual Studio 2008 - Required for FPU usageForF overview of camera bugs look at i f b l k thttp://studierstube.org/handheld_ar/camera_phones.php
  60. 60. Programming SymbianDevelopment tools Carbide.c++ Commercial version required for on-device debugging (important since emulator is bad…) SDK appropriate f your d for deviceMany quirks Crippled C i l d C++ support t Writing to static variables not allowed/recommended Cleanup Stack p API includes ~1500 classesMoving to Qt g
  61. 61. Programming iPhoneHarsh restrictions from Apple Apps have to go through the apps storeXcodeX d IDE for development f d l Nice development toolsObjective-CObjective C Required for application development Can call into C/C++ code C/CCamera API support in iOS 4+ Can overlay on live video y
  62. 62. AndroidHardware creators HTC, LG, Samsung, MotorolaWidelyWid l available phone il bl h Different form factors – tablets, phones, PC, etcMultiple versions and fragmentation Android 1.0, 1.6 Android 2.0, 2.1Free Tools Eclipse Development App Integrator
  63. 63. Mobile Graphics
  64. 64. Computer Graphics on Mobile Phones Small screen, limited input options Limited support for accelerated graphics Most phones have no GPU Mobile Graphics Libraries OpenGL ES ( , 2.0) p (1.0, ) - Cross platform, subset of OpenGL - C/C++ low level library Java M3G - Mobile 3D graphics API for J2ME platform - Object importer, scene graph library - Support from all major p pp j phone manufacturers
  65. 65. OpenGL O GL ESSmall-footprintSmall footprint subset of OpenGL OpenGL is too large for embedded devices!Powerful, lP f l low-level API full functionality for 3D games l l API, f ll f i li f Can do almost everything OpenGL can (but only one way) Available A l bl on all key platforms ll k l f Software and hardware implementations availableFullyF ll extensible ibl Extensions like in OpenGLNo redundancy! Convenience functions removed
  66. 66. OpenGL ES SLIDE 68 OpenGL ES vs. OpenGL (1.x) OpenGL OpenGL ES glBegin/glEnd 1 Primitive Types all no quads & polygons Data Types float, double int etc float double, int, etc… float, float fixed glDraw/Read Pixels glReadPixels only Textures T t 1D, 2D, 3D, cube 2D Stencil optional Window Bindings WGL, GLX, etc… EGL1: Except for Security Critical profile
  67. 67. OpenGL ES SLIDE 69OpenGL ES on mobile devices Java Applications C++ Applications Scenegraph API S h Game G Middleware Middl M3G (JSR 184) Engine Engines
  68. 68. OpenGL ES PerformanceFaster graphics (esp. hardware accelerated) ( )Longer batter performance (> 10%)
  69. 69. VersionsTwo major tracks Not compatible, parallel rather than competitiveOpenGL ES 1 x 1.x Fixed function pipeline Suitable for software implementations All 1.x are backwards compatibleOpenGL ES 2 x 2.x Vertex and pixel shaders using GLSL ES All 2.x will be backwards compatible 2x
  70. 70. Fixed Function (1 )F dF (1.x) http://www.khronos.org/opengles/2_X/
  71. 71. Programmable (2.x)P bl (2 ) http://www.khronos.org/opengles/2_X/
  72. 72. OpenGL ES 1.x vs 2.0
  73. 73. Tracking
  74. 74. Mobile Augmented Reality’s goalCreate an affordable, massively multi-user, widespread platform © Tinmith, U. of South Australia
  75. 75. Tracking is…Estimating the device‘s pose (position and orientation)Strictly in real time (30Hz) With high spatial precision (1cm, 1 degree) Robustly for operation by human user No unrealistic assumptions about HW Leaving enough power to other tasks (interaction graphics) (interaction,
  76. 76. Tracking requirements for AR on phones Fast and efficient Form factor: light and robust Track simultaneously A large number of objects By a large number of users Requires little or no … q Device modification Manual calibration (targeting non-technical users) ( g g ) Instrumentation of the physical environment Low costs
  77. 77. Tracking on mobile phonesVision-based tracking Marker-based tracking Model-based natural feature tracking Natural feature tracking in unknown environmentsSensor tracking GPS, inertial compass, gyroscope
  78. 78. Tracking for Handheld AR SLIDE 80 Backpack-based 1. Höllerer et al. (1999), Piekarski & Thomas al. (2001), Reitmayr & Schmalstieg (2003) ( ), ( ), y g( )Laptop, HMDEnhanced GPS (DGPS / RTK) + inertial sensor for viewpoint trackingHand tracking w/ fiducial markers
  79. 79. Tracking for Handheld AR SLIDE 81 Backpack-based 2. Kalkusch et al., 2002Video see-through HMD w/ camera gViewpoint Tracking w/ inside-out computer vision using markersARToolKit markers on walls installed and surveyed manually y y
  80. 80. Tablet PC / UMPC-based 1. Schall et al., 2006Hybrid tracking on UMPC Camera fiducial marker trackingg When no marker in view inertial sensor + UWB tracking
  81. 81. Tablet PC / UMPC-based 2. CAMERA LEDs Klein & Drummond, 2004Combining outside in (LED tracking for low accuracy robust pose) & outside-in accuracy,inside-out (edge-based tracking for high accuracy) computer vision
  82. 82. PDA-based 1.BatPortal (Newman et al., 2001) ( , ) SHEEP (MacWilliams et al., 2003) PDA as thin client (rendering & Tracking by ART (external IR tracking on server + VNC) cameras + retroreflective target) Ultrasonic tracking Projection-based AR environ.
  83. 83. PDA-based 2. Signpost on PDA (Wagner & Schmalstieg, 2003)First “truly” handheld AR platform: PDA + cameraStandalone, self-containedStandalone self contained AR systemOptimized fiducial marker tracking library
  84. 84. History of non-AR Tracking on Phones (1)AR-PDA (Gausemeier et al., 2003) Kick Real (Paelke et al., 2004) Model-based Model based tracking Edge detection of real foot + collision PDA = thin client detection w/ virtual ball tracking off-loaded to server 2D tracking and limited interaction Not real-time (tailored to game)
  85. 85. History of non-AR Tracking on Phones (2)PhoneGuide (Bruns et al., 2005) LightSense (Olwal, 2006) Neural network for recognizing visual External camera tracks cell phone LED features of museum artifacts Single user, only coarse position Combined w/ BT “tracker” tracking, no orientation Only object recognition O Border-case Border case of AR
  86. 86. History of non-AR Tracking on Phones (3)Mosquito Hunt (Siemens, 2003)Marble Revolution (BitSide, 2004)Pingis (VTT,Pi i (VTT 2006) TinyMotion (Wang et al., 2006)Game control w/ optical GUI control & input on cell phonesflow techniques w/ image differencing & block correlation
  87. 87. ARToolKit Tracking (Kato)ARToolKit - Computer vision based marker tracking librarieshttp://artoolkit.sourceforge.net/
  88. 88. History of AR Tracking on Phones (1)2003 ARToolKit on PDA Wagner et at.2004 3D Marker on Phone Möhring et al. g2005 ARToolKit on Symbian Henrysson et al.
  89. 89. Tracking for Handheld AR SLIDE 91Fiducial marker tracking on handheldsWagner et al., 2003 Möhring et al., 2004 Henrysson et al., 2006 Bucolo et al., 2005 Rohs, 2006
  90. 90. History of AR Tracking on Phones (2) 2005 Visual Codes Rohs et at at. 2008 Advanced Marker Tracking Wagner et al. 2008 Natural Feature Tracking Wagner et al.
  91. 91. What can we do on today‘s mobile phones? Typical specs 600+ MHz ~5MB of available RAM 160x120 - 320x240 at 15-30 Hz camera Possible to do Marker tracking in 5-15ms Natural feature tracking in 20-50ms
  92. 92. Handheld AR Interfaces
  93. 93. Handheld HCIConsider your userFollow good HCI principlesAdapt HCI guidelines for handheldsDesign to device constraintsRapid prototypingUser evaluation
  94. 94. Consider Your User■ They are probably mobile able to use the interface with one hand■ They want quick access to application content content. Want enhanced interaction with the real world Interaction with the real world is the main focus■ They expect to be able to multitask start phone call, use another application, etc p pp
  95. 95. Norman’s Principles of Good PracticeEnsure a high degree of visibility - allow the user to work out the current state of the system and the range of actions possible. f ti iblProvide feedback - continuous, clear information about the results of actions.Present a good conceptual model - allow the user to build up a picture of the way the system holds together, the relationships b h h l i hi between its different parts and h i diff d how to move from one state to the next.Offer good mappings g pp g - aim for clear, natural relationships between actions the user performs and the results they achieve.
  96. 96. High L l D i Guidelines Hi h Level Design G id liFrom Shneiderman’s 8 desktop design g p g guidelines: Enable Frequent Users to Use Shortcuts Offer Informative Feedback Design Dialogs to Yield ClosureGong and Tarasewich’s guidelines:G dT i h’ id li Design for Small Devices Design for Limited and Split Attention Design for speed and recovery Allow for personalization Design for Enjoyment
  97. 97. UI Device Constraints Comparing Desktop to Handheld Interfaces Screen Size Input Operation Multimedia ConnectivityDesktop > 1024 x 768 Mouse Two handed Millions of colours Wired Keyboard Stationary Graphics accel. Always On 5.1 AudioHandheld < 640 x 480 Stylus One handed 65K colours Wireless Touch Mobile No graphics accel. Maybe On Buttons Stereo audio
  98. 98. Example: E m l O2 Active M A ti MenuHighly visualUse PDA buttons for inputLarge icons and easy to read textVisually indicate which tabs are scrollableApplication UI looks different from device UI
  99. 99. iPhone GuidelinesMinimize required user input.Avoid unnecessary interactivity. y yProvide feedback when necessaryProvide fingertip sized target areas fingertip-sized areas.Avoid clutter and busy backgrounds.Express essential information succinctly. Make it obvious how to use your content. y
  100. 100. iPhone Interface
  101. 101. Designing for Device ConstraintsInput Device Touch, stylus, keyboard, buttons, keypad y y ypScreen Size, resolution ,Sensors Camera – frame rate image resolution rate, GPS – resolution, coverage Co pass accu acy Compass - accuracy
  102. 102. Sample Handheld AR InterfacesCleanLarge Video ViewLarge IconsText Overlay
  103. 103. Twitter 360www.twitter-360.com ppiPhone applicationSee geo-located tweets in real worldTwitter.comTwitter com supports geo tagging
  104. 104. Wikitude – www.mobilizy.com Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Bl hh Blah Bl Blah Bl h Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah
  105. 105. Information Filtering
  106. 106. Information FilteringInformation Filtering (Julier et al. ’00) al• Remove clutter by goal- and distance based filtering• User’s task is route finding: Sniper and relevant buildings are displayed;objects, which are determined to be unnecessary, removed bj t hi h d t i dt b d
  107. 107. HMD vs Handheld AR Interface Wearable AR W bl HandHeld AR Output: Display Input & Output Input
  108. 108. Handheld Interface MetaphorsTangible AR Lens Viewing Look through screen into AR scene Interact with screen to interact with AR content - Eg Invisible TrainTangible AR Lens Manipulation Select AR object and attach to device Use the motion of the device as input - Eg AR Lego
  109. 109. Handheld Display vs Fixed Display Experiment comparing handheld moving, to handheld button input, small fixed display, desktop display, large plasma Users performed (1) navigation task, (2) selection task Moving handheld display provided greater perceived FOV, higher degree of presence, faster completion timeJ. Hwang, J. Jung, G. Kim. Hand-held Virtual Reality: A Feasibility Study. In proceedings of VRST 2006
  110. 110. Search Task Completion Time
  111. 111. FOV, Presence and Immersion Perceived FOV and Actual FOV (deg. marked by subjects)70 64 58 6060 5250 4540 30 33 30 31 30 Perceived FOV30 Actual FOV2010 0 7 Motion Button Small 17 screen 42 screen 5.7 6 5.4 based hh based hh screen 5 4.7 4.7 4.9 4.3 4.5 4.3 4.3 4.1 4 Presence 3 Immersion 2 1 0 Motion Button Small 17 screen 42 screen based hh based hh screen
  112. 112. Rapid PrototypingSpeed development time by using quick hardware mockups p p y gq p handheld device connected to PC LCD screen USB phone keypad CameraCan use PC development tools Flash, Visual Basic, etc
  113. 113. Mobile Physical PrototypingBug Labshttp://www.buglabs.net/Open source hardware modules, eachpproducing one or more services. gModules snap together physically and theservices connect together logically to i h l i llenable users to easily build applications.
  114. 114. Software PrototypingPython Symbian (HIT Lab NZ) stbTracker wrapper Access to SMS, Bluetooth, GPS Rapid development
  115. 115. import e32 import appuifw from gles import *if e32.s60_version_info>=(3,0): import imp magnet=imp.load_dynamic(Magnet, t i l d d i (M t c:sysbinMagnet.pyd) bi M t d)else: import Magnetfrom Magnet import#Define Modeldef frameback(num_markers): if (num markers > -1): (num_markers 1): glMatrixMode(GL_PROJECTION)#Draw graphics…appuifw.app.orientation = landscape‘ # Use full frameSetCameraCallback(frameback) # Register callbackcreateCamera() # Define cameraInitGLES() # Start Open GLTrackerInit() # Start trackerInitCamera() # Start camera
  116. 116. Design GuidelinesApply handheld HCI guidelines for on screen content on-screen - large buttons, little text input, etcDesign physical + virtual interface elementsPick appropriate interface metaphor pp p p - “handheld lens” approach using handheld motion - Tangible AR for AR overlayBuild prototypesContinuously evaluate application
  117. 117. AR Browsers
  118. 118. AR BrowsersCommercial outdoor AR applications Junaio, Layar, Wikitude, etcAll have their own language specifications Wikitude – ARML Junaio - XMLNeed for common standard Based on existing standards for geo-located content etc Support for dynamic/interactive content Easier to author mobile AR applications Easy to render on AR browsers
  119. 119. Layar
  120. 120. Junaio
  121. 121. Hello World Exampleecho ""<?xml version="1.0" encoding="UTF-8"?> " " " "<results><poi id="1" interactionfeedback="none"> <name><![CDATA[Hello World POI]]></name> <description><![CDATA[[This is my first POI.]]></description> <l>48.1385,11.5750,0</l> , , <o>0,0,0</o><mime-type>text/plain</mime-type><thumbnail>http://www.junaio.com/publisherDownload/tutorial/icon.jpg</<thumbnail>http://www junaio com/publisherDownload/tutorial/icon jpg</thumbnail><icon>http://www.junaio.com/publisherDownload/tutorial/icon_map.png</icon></poi></results>"
  122. 122. KHARMA + Argon: A KML/HTML Architecture and Browser for AR Applications and Games Blair MacIntyre and Al Hill Bl i M I d AlexSchool of Interactive Computing, Georgia Institute of Technology 145
  123. 123. KHARMA:KHARMA KML/HTML Augmented Reality Mobile Architecture research tools { media hackers our past focus { skilled computationalists savvy technical designers Ygrasil IDE current focus { general public breadth of adoption Unity AR ToolkitDesigners AR Toolkit KHARMA 146
  124. 124. KHARMA: KML/HTML Augmented Reality Mobile Architecture Problem: limited authoring tools for mobile AR limited expression (coord, name, desc, link) vs higher hurdle of 3D no accepted standard and proprietary client protocol limited client-side interactivity is more akin to Web 1 0 1.0 Solution: HTML with KML combines What? and How? with Where? • allows extensive client side (albeit 2D) interactivity and expressivity • two the most broadly adopted standards for presentation and geo location • HTTP server distribution, CSS and Javascript allow for true Web 2.0 content + KML HTML 147
  125. 125. KHARMAArchitecture with four components Channel servers - delivering individual channels of AR content, Tracking servers - providing content related to location Tracking, Tracking infrastructure servers - delivering information about the physical environment, Mobile client - for generating the resulting augmentations
  126. 126. KML already supports HTML• description tag accepts CDATA enclosed markup - but no global styling of scripting support• no control over balloon styling• no control over balloon position and orientation• no relative positioning <Placemark id="culc_center"> <name>CULC Visualization</name> <description><![CDATA[ <div id="culc_image">Georgia T h> <di id " l i ">G i Tech> <img src="http://www.culc.net/culc.png"></div> </description> <Balloon> <location> <latitude>34.0</latitude> <longitude>--84.5</longitude> </location> <orientation> <heading>31</heading> </orientation> </Balloon> </Placemark>
  127. 127. KARML extension to KML <Placemark id="culc_center"> <name>CULC Visualization</name>• added undecorated displayMode <description><![CDATA[ <div id="culc_image"><img src="http://www.culc.net/culc.png"></div> </description> <Style>• added Balloon element <BalloonStyle> <displayMode>undecorated</displayMode> - similar in nature to Model element </BalloonStyle> </Style> <Balloon> <locationMode targetId=”culc_g g geospot”>relative</locationMode> p• relative locationMode <location> - accepts “units” attribute <latitude units="meters">4.0</latitude> <longitude units="meters">-2.5</longitude> </location> <orientation> <heading>31</heading> </orientation> </Balloon> </Placemark> 151
  128. 128. GPS Located Content <Placemark>Geospot <name>GeoSpot</name> <description>a surveyed GeoSpot </description> Surveyed location <Camera> <!-- camera element for GeoSpot --> p Moves camera to <longitude>-84.394135</longitude> <!- GPS coordinates --> <latitude>33.76083</latitude> fixed location <altitude>0</altitude> <TimeStamp> <! when GeoSpot was surveyed --> <!-- > <when>1997-07-16T10:30:15+03:00</when> </TimeStamp> <Icon> <href>http://geospot.imtc.gatech.edu/image/03_icon.png</href> </Icon> </Camera> <Point> <! location displayed on map or within browser view --> <!-- > <coordinates>-84.394135,33.76083</coordinates> </Point> </Placemark>
  129. 129. For more information, please visit: http://www.argon.gatech.edu/ http://www argon gatech edu/
  130. 130. Developing AR Experiences Sony CSL © 2004
  131. 131. Game Case Study
  132. 132. Resources
  133. 133. ResourcesBooksMobile Interaction Design Matt J M Jones and Gary Marsden dG M dDesigning for Small Screens Studio 7.5Handheld Usability Scott WeissDesigning the Mobile User Experience Barbara Ballard
  134. 134. Developer Guidelines Palmhttp://www.access-company.com/developers/documents/docs/ui/UI_Design.html Zen of Palm guidelineshttp://www.access-company.com/developers/documents/docs/zenofpalm.pdf Motorolahttp://developer.motorola.com/docstools/developerguides/ iPhone Human Interface Guidelineshttp://developer.apple.com/documentation/iPhone/Conceptual/iPhoneHIG/
  135. 135. Handheld HCI Design WebsitesDo’s and Don’ts of PocketPC designhttp://www.pocketpcmag.com/_archives/Nov04/Commandements.aspxUsabilityU bili special i i l interest group – h dh ld usability handheld bilihttp://www.stcsig.org/usability/topics/handheld.htmlUsable Mobile websitehttp://www.smartgroups.com/groups/usablemobileMobile Coders Websitehttp://www.mobilecoders.com/Articles/mc-01.asphttp://www mobilecoders com/Articles/mc-01 aspUniv of Waikato Handheld Grouphttp://www.cs.waikato.ac.nz/hci/pdas.htmlMobile Interaction Websitehttp://www.cs.waikato.ac.nz/~mattj/mwshop.html
  136. 136. Platform – Recommended readingLots of low level informationon the complete ARM family p yValuable tool for driver andframework developersNot that important for pureapplication developers
  137. 137. Platform – Recommended readingVery low level and targeted for PCsMost information outdated on PCEffective memory usage oneof the most importantoptimization strategieson mobile devices!
  138. 138. Tracking – Recommended readingLots of the basics on theComputer Vision you will p yneed for AR trackingSeveral code and pseudo-code pseudo codesnippets
  139. 139. Tracking – Recommended readingAll about the geometryyou will need fora tracking system Camera models Projection Epipolar geometry Homographies H hi …
  140. 140. Graphics – Recommended reading Mobile 3D Graphics (all b t O GL ( ll about OpenGL ES 1.x) 1 ) OpenGL ES 2.0 Programming G Guide OpenGL ES 2.0 Man Pages http://www.khronos.org/opengles/sdk/docs/man/ ShaderX7 Chapter on “ Augmented Reality on Mobile Phones”
  141. 141. OpenGL ES ResourcesKhronos Group OpenGL ES Page http://www.khronos.org/opengles/OpenGL ES 2.0 Book http://www.opengles-book.com/AMDs OpenGL ES 2.0 Emulator
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×