Designing Mobile AR Applications

5,338 views

Published on

Workshop taught by Mark Billinghurst at the ICIDM 2013 conference about using the Junaio platform for designing mobile AR applications. Presented on December 2nd 2013. Provides material about using Metaio Creator, and Junaio coding for developing marker based and GPS based mobile AR applications.

Published in: Technology, Business

Designing Mobile AR Applications

  1. 1. Designing Mobile AR Applications Mark Billinghurst mark.billinghurst@hitlabnz.org The HIT Lab NZ, University of Canterbury December 2nd 2013
  2. 2. Introduction
  3. 3. 1983 – Star Wars
  4. 4. 1999 - HIT Lab US
  5. 5. 1998: SGI O2 CPU: 300 Mhz HDD; 9GB RAM: 512 mb Camera: VGA 30fps Graphics: 500K poly/sec 2008: Nokia N95 CPU: 332 Mhz HDD; 8GB RAM: 128 mb Camera: VGA 30 fps Graphics: 2m poly/sec
  6. 6. Mobile Phone AR   Mobile Phones   camera   processor   display   AR on Mobile Phones   Simple graphics   Optimized computer vision   Collaborative Interaction
  7. 7. 2005: Collaborative AR   AR Tennis         Shared AR content Two user game Audio + haptic feedback Bluetooth networking
  8. 8. Mobile AR History
  9. 9. Evolution of Mobile AR Camera phone Wearable Computers Camera phone - Thin client AR Wearable AR Handheld AR Displays Camera phone - Self contained AR PDAs -Thin client AR PDAs -Self contained AR 1995 1997 2001 2003 2004
  10. 10. Handheld Displays Tethered Applications   Fitzmaurice Chameleon (1994)   Rekimoto’s Transvision (1995)   Tethered LCD   PC Processing and Tracking
  11. 11. Handheld AR Display - Tethered 1995, 1996 Handheld AR   ARPad, Cameleon   Rekimoto’s NaviCam, Transvision   Tethered LCD   PC Processing and Tracking
  12. 12. Mobile AR: Touring Machine (1997)   University of Columbia   Feiner, MacIntyre, Höllerer, Webster   Combines           See through head mounted display GPS tracking Orientation sensor Backpack PC (custom) Tablet input
  13. 13. MARS View   Virtual tags overlaid on the real world   “Information in place”
  14. 14. Backpack/Wearable AR 1997 Backpack AR         Feiner’s Touring Machine AR Quake (Thomas) Tinmith (Piekarski) MCAR (Reitmayr)   Bulky, HMD based
  15. 15. Mobile AR - Hardware RTK correction Antenna GPS Antenna HMD Controller Example self-built working solution with PCI-based 3D graphics PCI 3D Graphics Board Tracker Controller PC104 Sound Card DC to DC Converter Wearable Computer CPU PC104 PCMCIA Battery GPS RTK correction Radio Hard Drive Serial Ports Columbia Touring Machine
  16. 16. HIT Lab NZ Wearable AR (2004)   Highly accurate outdoor AR tracking system   GPS, Inertial, RTK system   HMD   First prototype   Laptop based   Video see-through HMD   2-3 cm tracking accuracy
  17. 17. Image Registration AR Stakeout Application
  18. 18. Wearable AR Video
  19. 19. Sharp J-SH04   1997 Philip Kahn invents camera phone   1999 First commercial camera phone
  20. 20. Millions of Camera Phones 1200 1000 800 DSC 600 Phone 400 200 0 2002 2003 2004 2005 2006 2007 2008 2009 2010
  21. 21. Handheld AR – Thin Client 2001 BatPortal (AT&T Cambridge)   PDA used as I/O device   Wireless connection to workstation   Room-scale ultrasonic tracking (Bat) 2001 AR-PDA (C Lab)   PDA thin graphics client   Remote image processing   www.ar-pda.com
  22. 22. Mobile Phone AR – Thin Client 2003 ARphone (Univ. of Sydney)   Transfer images via Bluetooth (slow – 30 sec/image)   Remote processing – AR Server    
  23. 23. Early Phone Computer Vision Apps 2003 – Mozzies Game - Best mobile game Optical motion flow detecting phone orientation Siemens SX1 – Symbian, 120Mhz, VGA Camera 2005 – Marble Revolution (Bit-Side GmbH) Winner of Nokia's Series 60 Challenge 2005 2005 – SymBall (VTT)
  24. 24. Handheld AR – Self Contained 2003 PDA-based AR   ARToolKit port to PDA   Studierstube ported to PDA   AR Kanji Educational App.   Mr Virtuoso AR character   Wagner’s Invisible Train -  Collaborative AR
  25. 25. Mobile Phone AR – Self Contained 2004 Mobile Phone AR   Moehring, Bimber   Henrysson (ARToolKit)   Camera, processor, display together
  26. 26. AR Advertising   Txt message to download AR application (200K)   See virtual content popping out of real paper advert   Tested May 2007 by Saatchi and Saatchi
  27. 27. 2008 - Location Aware Phones Motorola Droid Nokia Navigator
  28. 28. Real World Information Overlay   Tag real world locations   GPS + Compass input   Overlay graphics data on live video   Applications   Travel guide, Advertising, etc   Eg: Mobilizy Wikitude (www.mobilizy.com)   Android/iOS based, Public API released   Other companies   Layar, AcrossAir, Tochnidot, RobotVision, etc
  29. 29. Layar – www.layar.com
  30. 30. 2013 State of the Art Handheld Hardware available PDA, mobile phones, external cameras Sensors: GPS, accelerometer, compass Software Tools are Available Tracking: ARToolKitPlus, stbTracker, Vuforia Graphics: OpenGL ES Authoring: Layar, Wikitude, Metaio Creator What is needed: High level authoring tools Content development tools Novel interaction techniques User evaluation and usability
  31. 31. Mobile AR Companies   Mobile AR   GPS + compass   Many Companies               Layar Wikitude Acrossair PressLite Yelp Robot vision Etc..
  32. 32. $2 million USD in 2010 $732 million USD in 2014
  33. 33. Qualcomm           Acquired Imagination October 2010 - Releases free Android AR SDK Computer vision tracking - marker, markerless Integrated with Unity 3D renderer http://developer.qualcomm.com/ar
  34. 34. Rock-em Sock-em   Shared AR Demo   Markerless tracking
  35. 35. Mobile AR Browsing
  36. 36. Nokia City Lens   More recent AR Browser
  37. 37. AR Browsers   AR equivalent of web browser   Request and serve up content   Commercial outdoor AR applications   Nokia, Junaio, Layar, Wikitude, etc   All have their own language specifications   Wikitude – ARML   Junaio – XML, AREL
  38. 38. Architecture
  39. 39. Junaio - www.junaio.com
  40. 40. Key Features   Content provided in information channels   Over 2,000 channels available   Two types of AR channels   GLUE channels – visual tracking   Location based channels – GPS, compass tracking   Simple to use interface with multiple views   List, map, AR (live) view   Point of Interest (POI) based   POIs are geo-located content
  41. 41. Try it Yourself   Download Junaio (app store, android market)   Search for Junaio   Run Junaio   To try Glue channels   Download the Junaio Demo Book from www.junaio.com   Hit scan button, point at QR codes   To try Location channels   Hit search icon in Junaio   Click popular tab, pick channel (eg “Flickr”)
  42. 42. Junaio Interface (Location Based) Search Icon AR Tag Current Channel View Mode Scan Mode Radar Display
  43. 43. Selecting an AR Tag   Selecting a POI shows more information
  44. 44. Multiple Views Map View List View AR (Live) View   Select View Mode to see different views
  45. 45. QR Code Launch   Hit scan button on interface, point at QR code
  46. 46. Glue Tracking - Markerless   Search for “instant tracker”
  47. 47. How Junaio Works
  48. 48. Back-end Servers
  49. 49. Data Flow
  50. 50. Creating Your First Glue Channel (Using Junaio Creator)
  51. 51. Key Steps 1.  Download and install Junaio application   From iOS and Android app stores 2.  Create a developer account on Junaio.com 3.  Download and install Metaio Creator   From http://www.metaio.com/creator/ 4.  Build sample AR scene 5.  Test scene on desktop 6.  Publish mobile AR channel
  52. 52. Junaio.com Click here to download mobile application Click here to create developer account
  53. 53.   http://www.junaio.com/develop/
  54. 54.   http://www.metaio.com/creator/
  55. 55. Creator Features           Drag and drop AR scene creation Multiple types of content (2D, 3D, text, video, etc) Multiple tracking techniques (image, model, point cloud) Upload to Metaio server space Offline preview mode
  56. 56. Creator Interface
  57. 57. Creator Interface Resources UI Tools AR view Chanel Info Tracking Images
  58. 58. Demo 1.  2.  3.  4.  5.  Upload tracking image Add 3D content Manipulate content Test scene Publish scene
  59. 59. Desktop Test   Click “Quick Preview” button
  60. 60. Creating the AR Channel   Click “New Chanel”, Fill out Channel Details
  61. 61. Testing the Channel   Launch Junaio, scan QR code
  62. 62. QR Code Launch   Hit scan button on interface, point at QR code
  63. 63. Test Result
  64. 64. Other Demos   Adding other trackable images   Adding other content (video, 3D models, etc)   Adding buttons and interactions   Using the User Interface designer
  65. 65. More Trackables   Add more images   Set properties
  66. 66. Other Types of Trackables   Image, object, environment tracking   Use Metaio Toolbox for data capture
  67. 67. Good Tracking Patterns   pattern that is highly structured   lot of visual hints with different colors   high contrasts and sharp edges   pattern in a "common" format,   Square or rectangle format in 3:2 or 4:3 or similar   not too dark and no reflection points   shortest side of the image 150 – 200 pixels
  68. 68. Good Examples
  69. 69. Bad Tracking Patterns   Reference Image not flat and blurry   Shadows create false contrasts    Angled reference images create false reference orientation   Pattern too bright or dark   Angled with surrounding information
  70. 70. Bad Examples
  71. 71. Adding More Content   3D models   Position   Adjust properties -  Shadow, occlusion, etc -  Trigger animation   Video content   In-page video, full-screen video   Websites, audio   Page triggered
  72. 72. Adding Buttons   Drag button icon into scene   Scale and position   Right click to add behaviours   Open website, play video, etc
  73. 73. Using the User Interface Designer   Drag and drop 2D UI elements   Buttons, images, etc   Runs in screen aligned mode
  74. 74. Making the Chanel Public Click to change channel status   Becomes available for anyone in the world   Takes 1-2 days for review by Metaio   Need a paid Creator License ($530 USD)
  75. 75. More Information   http://dev.metaio.com/creator/   Online tutorials, getting started, helpdesk, etc
  76. 76. Creating Your First POI Channel (Creating by Hand)
  77. 77. You will Need   A Junaio developer account   Create at dev.junaio.com   A web server where content can be uploaded   Eg free server from http://www.000webhost.com/   GPS Location of POI   POI content   Text, 2D image, etc
  78. 78. Free Webserver   Other options also available
  79. 79. Finding the POI location   Use Google maps, right click the POI location   Copy Lat/Long information
  80. 80. Server Content   Download “Hello World” template content   http://www.junaio.com/develop/quickstart/   Edit on local machine   Edit index.php to add POI information   Use own POI icon   Upload to web server
  81. 81. Edit index.php POI icon POI location POI pop-up info   Edit index.php to add your own POI information
  82. 82. Create a New Channel   Click ‘My channels’ then ‘Create a New Channel’
  83. 83. Fill Out the New Channel Form
  84. 84. Channel Creation   Use any name and channel description   Channel Type: Location Based Channel   Callback URL is most important   Path to the index.php file on your server   http://www.junaiotest.comze.com/JunaioTest/1HelloWorld/?path=   Note ?path= at end of URL, you may not need this   Once channel is saved then it is added to My Channels list   Next Validate the Channel
  85. 85. Channel Validation   Runs a number of tests to see if server path is correct, if content is there, returned code correct
  86. 86. Validation Tests
  87. 87. Expand Results – shows XML feedback
  88. 88. Loading the Channel   If the Channel is validated you can run it   Either login into Junaio and enter developer mode   Channel appears under ‘My Favourites’   Or use QR code from My Channel page
  89. 89. AR Outcome
  90. 90. XML Parameters   Many XML Parameters can be set   See http://www.junaio.com/develop/docs/arel-xml-schemaxml-parameters/
  91. 91. Adding More POI – edit index.php <results> <object id="1”> //Define POI One <location> <lat>-43.536743</lat> <lon>172.587705</lon> <alt>0</alt> </location> </object> <object id="2"> //Define POI Two <location> <lat>-43.536743</lat> <lon>172.587000</lon> <alt>0</alt> </location> </object> </results>"; POI One Location POI Two Location
  92. 92. Junaio Results
  93. 93. Limitations of Plain XML   No interactivity   Only simple pop-ups   No user interface Customizations   Can only use Junaio GUI elements   No local interactivity   Always needs remote server connection
  94. 94. Junaio AREL
  95. 95. AREL   Augmented Reality Environment Language   Overcomes limitations of XML by itself   Based on web technologies; XML, HTML5, JavaScript   Core Components 1. AREL XML: Static file, specifies scene content 2. AREL JavaScript: Handles all interactions and animation. Any user interaction send an event to AREL JS 3. AREL HTML5: GUI Elements. Buttons, icons, etc   Advantages   Scripting on device, more functionality, GUI customization
  96. 96. Example 2: Customizing your POI   Using AREL HTML5 to develop custom interface   Download Tutorial 2   http://www.junaio.com/develop/quickstart/customizingyour-pois-images-sounds-videos-and-more/   In Example 1 edited index.php, now use search.php
  97. 97. index.php   Just loads search.php
  98. 98. search.php   Loads the AREL helper classes   php code providing valid information to Junaio   Contains Channel Definition 1. Start it startArelXMLHelper::start(NULL, WWW_ROOT . "/arel/index.php"); 2. Return objects ArelXMLHelper::outputObject($oObject); 3. End it ArelXMLHelper::end();
  99. 99. Defining a POI //1. Sound POI $oObject = ArelXMLHelper::createLocationBasedPOI( "1", //id "Hello Sound POI", //title array(48.12310, 11.218648, 0), //location WWW_ROOT . "/resources/thumb_sound.png", //thumb WWW_ROOT . "/resources/icon_sound.png", //icon "This is our Sound POI", //description array(array("Start Audio", "soundButton", "http://www.junaio.com/publisherDownload/ tutorial/test.mp3")) //buttons ); //output the object ArelXMLHelper::outputObject($oObject);
  100. 100. Location Based POI
  101. 101. AR Application Running
  102. 102. Loading a 3D model
  103. 103. Example 3: Loading a 3D Model   Position a model relative to the user position if(!empty($_GET['l'])) $position = explode(",", $_GET['l']); //calculate the position of T-Rex based on the position of the request. An offset is added to the latitude value. $tRexLocation = $position; $tRexLocation[0] += 0.00004; Use createLocationBasedModel3D to load 3D model
  104. 104. createLocationBasedModel3D
  105. 105. Loading T-Rex //T-Rex as static obj $oObject = ArelXMLHelper::createLocationBasedModel3D( "trex", //ID "The T-Rex", //name "http://dev.junaio.com/publisherDownload/junaio_model_obj.zip", //model NULL, //texture $tRexLocation, //position array(5,5,5), //scale new ArelRotation(ArelRotation::ROTATION_EULERDEG, array(0,-90,0)) // rotation );
  106. 106. Result
  107. 107. 3D Models   Junaio supports two model formats:   MD2: Animated models, simple textures   OBJ: Static models, high quality textures   Use OBJ for high quality static models, MD2 for animated   Making Models   Make Models using Blender or similar tools   May need file conversion tools   Limit size to 500 – 1000 polygons/model   See http://www.junaio.com/develop/docs/3d-models/
  108. 108. Adding Interactivity
  109. 109. Basic Interactivity   Add a button on screen to move virtual character   Use the following   HTML: button specification   Javascript: Interaction   PHP/XML: 3D model   Junaio Tutorial 5   http://www.junaio.com/develop/quickstart/advancedinteractions-and-location-based-model-3ds/
  110. 110. Server File Structure HTML – GUI JavaScript - interactivity Main Index PHP - content
  111. 111. search.php – specify Lego Man if(!empty($_GET['l'])) $position = explode(",", $_GET['l']); Use local position … //return the lego man $oLegoMan = ArelXMLHelper::createLocationBasedModel3D( "1", // id Lego model and texture "lego man", //title WWW_ROOT . "/resources/walking_model3_7fps.md2", // mainresource WWW_ROOT . "/resources/walking_model.png", // resource $position, // location array(0.2, 0.2, 0.2), // scale new ArelRotation(ArelRotation::ROTATION_EULERRAD, array(1.57,0,1.57)) // rotation ); …
  112. 112. styles.css – HTML GUI #buttons { position: absolute; bottom: 44px; right: 44px; } Button location .ipad div { width: 104px; height: 106px; } #buttons div { background-image: url("../images/button.png"); background-repeat: no-repeat; background-size: 100%; float: left; } Button style
  113. 113. Logic_LBS5.js - JavaScript   Create an event listener   setEventListener();   Add functionality to model object   Load model from scene   Adding model behaviours   Add functionality to GUI objects   Define the event listener   Bind model behaviours to GUI objects
  114. 114. Result
  115. 115. Looking to the Future What’s Next?
  116. 116. MIT Wearable Computing (1996)
  117. 117. Google Glass
  118. 118. What's Inside Google Glass?
  119. 119.   Hardware   CPU TI OMAP 4430 – 1.2 Ghz   16 GB SanDisk Flash,1 GB Ram   570mAh Battery   Input   5 mp camera, 720p recording, microphone   GPS, InvenSense MPU-9150 inertial sensor   Output   Bone conducting speaker   640x360 micro-projector display
  120. 120. View Through Google Glass Always available peripheral information display Combining computing, communications and content capture
  121. 121. User Interface   dfasdf
  122. 122. User Experience   Truly Wearable Computing   Less than 46 ounces   Hands-free Information Access   Voice interaction, Ego-vision camera   Intuitive User Interface   Touch, Gesture, Speech, Head Motion   Access to all Google Services   Map, Search, Location, Messaging, Email, etc
  123. 123. Junaio on Google Glass   Junaio Mirage - Junaio Browser on Glass   Adapt UI to Glass (see through, touch gesture)   Backend server unchanged
  124. 124.   asdfa
  125. 125. Living Heads Up vs. Heads Down
  126. 126. Competitors   Vuzix M100   $999, profession   Recon Jet   $600, more sensors, sports   Opinvent   500 Euro, multi-view mode   Motorola Golden-i   Rugged, remote assistance
  127. 127. Projected Market   > 10 million displays by 2016
  128. 128. Resources
  129. 129. Book Demos
  130. 130. Resources •  Developer documentation –  http://www.junaio.com/develop/docs/ •  Google Group –  https://groups.google.com/forum/?fromgroups#!forum/junaiodeveloper
  131. 131. Conclusions
  132. 132. Conclusion   Junaio provides easy way to create mobile AR   Cross platform, client/server based, customisable   Metaio Creator allows non-programmers to create mobile AR applications   Drag and drop visual interface   Junaio supports HTML, XML, Javascript   AREL format   Support for wearable devices/computers   Google Glass, etc
  133. 133. More Information   Mark Billinghurst   mark.billinghurst@hitlabnz.org   HIT Lab NZ   http://www.hitlabnz.org/

×