Future of Mobile Augmented Reality (Zenitum's View Point)


Published on

Future of Mobile Augmented Reality
- Zenitum's internal view point and the ultimate goal

Published in: Technology, Business
1 Comment
  • CEO님 자료 감사합니다. 3/16일에 KAIST에서 인사드렸던 삼성전자 문영호책임입니다. 앞으로도 AR의 미래에 대한 고민에 동참하도록 하겠습니다. 감사합니다
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Future of Mobile Augmented Reality (Zenitum's View Point)

  1. 1. March 2012Albert Heekwan KimFounder & CEOmail : albertk@zenitum.comtwitter : @hkalbert
  2. 2. Market - Vision AR Market will be Gigantic predicted by market research firms around the world ABI Research said We say... Gazillion dollar Market!! $350 million in 2014 Juniper Research said $732 million in 2014 ARC Chart said $2.2 billion in 2015* It could be worth 1,000 time s more!* Mobile AR application market size in US alone 2
  3. 3. Market - Vision Evolution of Computers Computing paradigm evolved to process more informationMore computing power is needed to process the era ofhighly realistic visual data for pleasing human eyes Visual Computing Most power goes to • Highly realistic graphics • precision vision tracking • visual representation of data 3
  4. 4. Market - Vision AR naturally mixes Real & Virtual thus people intuitively understand the world digitally Tablet Devices AR GogglesAll you need to do is to look at the objects! Bionic eyes for everyone!! 4
  5. 5. Market - Vision people play games inside of Mixed Worlds where the virtual & the real co-exist seamlesslyCoca Cola Commercial (2009) Real & virtual are all mixed in the Real World (within its physical frames) 5
  6. 6. Market - Enabling the Vision the ultimate AR tracking technology must compute the 3D coordinates from the real world Mixed Reality on to the real world real world 3D coordinates & objects detection Street seen by naked eyes Application Life-size virtual Processor Ninja on the street! Visual information SCREEN CAMERA Virtual Wireless Contextual Information Contents Sensors Users (location, environment, etc) Real/Physical World (Street of Kyoto, Japan) Mixed Reality System Mixed Realities via the ultimate visual tracking technology 6
  7. 7. Market - Status are we there yet? NO! but somewhat stable technology is available now GPS-only Tracking NFT1 - only Tracking Panoramic Scene with GPS Small Target Tracking (Vision) Large Location Tracking (GPS) • Tracking error rate : 50~90m • Precise visual tracking on targets • GPS for absolute position on the Earth • AR objects never align with real scene • Only tracks objects near by • Specific target tracking with vision • Revisit rate is extremely low! • Targets must be processed prior to use • Combine tracking for large & small • Tracking the viewer, NOT the targets • Targets only. Can’t track 3D space • only works from specific view points No practical uses! Commercially available! R&D phases, yet1. Natural Feature Tracking (NFT) : Realtime visual tracking of registered texture patterns on planar surfaces (i.e. magazines, photos, etc) 7
  8. 8. Market - Problems limitations on current NFTs1 1. they always need “pre-processing2” of targets Target Pre-Processing on NFT Natural Feature Tracking 1. Select a target image Best to use graphical images (i.e. photos) Preferably jpg or png file Takes 2. Extract feature points 30-40 min. Extracting feature points based on to generate proprietary algorithm a target info Target homography 344, 2334, 34,2223, 45, 45 3. Calculate a “target info” 343434, 4454, 332, 444, 5 4533, 22, 1,2 33 Calculating images from many different angles, and out put “target information” And then, AR App Development • Precise visual tracking on targets Develop apps with “target info” • Only tracks objects near by With the “target info,” developers now • Targets must be processed prior to use able to code AR apps with “target info”1. Zenitum, Qualcomm and Metaio provides NFT technologies. They all share similar problems.2. Pre-processing is required to create a target information out of physical objects. (Training or pre-define targets also used in this document as the same meaning) 8
  9. 9. Market - Problems limitations on current NFTs 2. No more than TEN targets per application Targets per applications Up to 10 targets Up to 7 targets Up to 7 targetsARAD Client App (2011) - Only able to track up to Seven targets simultaneouslyBusiness Models with1) FEW targets simultaneously• AR games (with less than 10 cards)• AR Advertisements (with less 10 targets)2) Thousands targets simultaneously• Massive Ads on entire product lines (i.e. P&G products)• Interactive Ads & games (i.e. Games on entire Kelloggs cereal boxes)• AR books for entire publishings (i.e. Video lecture on entire Penguin Classics)• Massive AR search (i.e. Dietary video info. on all products in supermarkets)• AR knowledge search (i.e. AR contents on books in libraries) the user clicks the virtual button, then related on a book shelf, relation lines pop up when• and more web pages pops up next to the book pulling out a book. 9
  10. 10. Market - Steps into Mixed Realty overcoming techno-barriers then evolve into new paradigm : Mixed Reality NOW Solutions Target Pre-processing1 Allowing only Few Targets2 (~2011) but Problems • annoyance to developers & users • annoyance to developers & users • limiting business models around AR • limiting business models around AR TO BE Interim Large number of Instant Target Generation (2011~) Solution • Instant target generation by either developers or users • allowing vast number of targets to cover many business models GOAL Ultimate Natural Scene Tracking (2013~) Solution • understands the physical space & generating 3D coordinates in real-time • mixing virtual data on the physical space for mixed reality interaction Mixed Reality - Visually indistinguishable mix of Real & Virtual1. refer p. 7 for more on “Target Pre-processing” 2. refer p. 8 for more on “Tracking few targets” 10
  11. 11. Market - Macro AR Technology Trends To Reach AR Event Horizon Two Major Infrastructure Must be completed Massive AR Content Creation & Distribution Instant (on-line) Target Generation Massive Target Search (preferably video streaming search) Natural Scene Tracking Hardware Innovation (GPGPU, AR Core, Multi-Cameras) COTS Devices (AR Goggles) 11
  12. 12. Market - At the end of the day AR market will be explode! as the technology matured More!! Natural Scene Tracking AR as the ultimate HCI!! 1st Wave 2nd Wave GPS based tracking NFT with target training Early adopters & hobbyists Real business applications 3rd Wave Instant Target Generation Tracking many objects Market Size New applications (i.e. SNS) 2008 2010 2011 2012 12
  13. 13. Zenitum, NOW - Technology AR Tracking Engine #1 : zFT Pre-assigned targets tracking with NFT Performance : 15 ~ 30 frames per second Min. CPU requirement : ARM11 / 700 MHz or higher Pre-assigning a target takes about 30 minutes Max. number of targets per application : 10 (Recommends 7) Platforms supported : iOS, Android, Win7, MacOS Zenitum Feature Tracker (zFT) is a natural feature tracking Multiple target combination available engine that tracks 3D movement of planar targets(i.e. Some portion of zFT cab be embeddable to microchips books, magazines, product packages, etc) (tested on Altera Stratix II FPGA DSP) zFT is commercially available NOW. Available Now! 13
  14. 14. Zenitum, NOW - Technology AR Tracking Engine #2 : AR Goo Instant target generation and tracking untrained object (a ceramic coaster) Placing AR content instantly!!! click! TM AR Googeneration AR Tracker Instant Target 2nd generation NFT engine with no target pre-assigning Targets can be generated instantly on user Max. number of targets per an application : 5 Engine available for iOS, Android and PC platform Some portion of ARGoo, zarCORE is available for microchips (tested on Altera Stratix II FPGA DSP) solving problem #1 14
  15. 15. Zenitum, NOW - Technology Massive Target Search Engine : zARTIS fast target image search and find stream of images AR Contents Content AR Streamer DB Service Server zARTIS Zenitum zARTIS 1) search result AR Target DB (target info.) Image Search 2) AR contents TMzARTISimage search enginemassive target Works with either zFT or ARGoo tracking engines zARTIS support for unlimited targets (up to 15,000 targets per channel) Each application can have unlimited number of channels Content builders easily create targets and AR contents through web interface solving problem #2 15
  16. 16. Zenitum, NOW - Technology AR Tracking Engine : D-Track 1st attempt for the space & depth tracking Initial depth calculation by users’ cam movement A planar 3D filed is generated for 3D contents mix D-Track I LE D! FAD-Track TM Space and Depth tracking #1 How to initialize R&D from May 2008 ~ Sept. 2010 1) Run the App with D-Track Oxford’s “PTAM” licensed algorithm based 2) Swing the device Mixture of PTAM tracker and zFT Detector - left & right Implemented on Samsung Omnia II with Windows Mobile - back & forth Every initialization result is differ from one another 3) Generating 3D planar coordinate 30% of initialization failed miserably!!Heavily depended on the initialization process Not ROBUST enough to be commercialized 16
  17. 17. Zenitum, NOW - Technology AR Tracking Engine #3 : SceneTrack finally, mother of all tracking for Mixed RealityTracking starts with one object 3D coordinate extended 3D coordinate extended (3D coordinate) Gradually includes neighboring space TM SceneTrack Robust Space & Depth tracking R&D since Sept. 2010 ~ Mixture of AR Goo engine & SfM (Structure from Motion) How to initialize Start with one object (i.e. magazine, photo, or product pkg) 1) Run the App with SceneTrack The tracking space gets larger as the camera moves 2) Choose an image as an initial position 3) From the image, move the camera to For max. performance, users can add images in the space generate 3D coordinate of larger space as 3D coordinate stepping stones 4) More images can be selected & work as stepping stones for larger space trackingEvery user has exact same virtual coordinate the interim solution 17
  18. 18. Zenitum, NOW - Technology AR Tracking Engine #4 : zarCORE direct hardwired engine with steroid 3rd Party APP APP APP APP APP APP Applications AR Goo Algorithm ARGoo SDK Application APIs Selected algorithms 3D Renderer Game Engine AR Goo zarCORE Android iOS Windows Phone, etc Application Processors Direct Access to zarDrive Hardware side Software side TM Selected high speed computation algorithms from ARGoo zarCORE Directly hardwired onto application processors Boosting whole performance in 150 folds IP Core for Application Processors - 150 times faster calculation - Very precise tracking Leaving CPU for other computationally rich tasks - Realistic 3D Graphics Rendering - Scene Tracking (if developers use SceneTrack) 18
  19. 19. Zenitum, NOW - Technology AR Tracking Engine Evolution perspiration, despair and triumph ARGoo zarCORE SceneTrack 2.0~ zARTIS 1.0~ 2.0~ 1.0~2012 zarCORE SceneTrack 1.02011 zFT ARGoo 4.0 1.0 zARTIS SceneTrack Core inheritance2010 ARGoo Non-core inheritance D-Track 1.0 Depth Tracking (Open source based)2009 zFT 2.0~3.0 D-Track Open Source Tracking engine licensed2008 ARCORE Natural Scene Tracking 1.0 miniPTAM zFT NFT (Instantly created target) 1.0 ARCORE PTAM2007 NFT (pre-processed target) zFT Marker based Tracker2006 Frozen MobEYE Abandoned2005 Marker based Markerless Instant Target Generation AR IP Core Natural Scene Tracking 19
  20. 20. Zenitum, FUTURE - Evolution Gradual Shift to B2C Market Current AR technology is not for the mass users lems s olved!! ob PrTechnology (to be) AR Technology (now) AR Tracks only tiny target space Tracks the real 3D spaces (i.e. magazine pages, street billboards) (i.e. desktops, rooms, buildings, streets) Tracks less than 10 targets Tracks many targets at any given time (i.e. 10 cards, 10 pages of Magazine) (i.e. AR contents on products on aisles of super markets) Every target must be pre-processed Generate targets in real-time (average target processing time : 30 min./target) (pointing objects to generate targets in real-time) Novelties & Promotional Mass user ready! Sample demo AR applications Zenitum already solved problem #3 Simple mobile AR games with less than 10 targets and partly solved problem #2 and #1 Mobile AR advertisements with less than 10 targets 20
  21. 21. Zenitum, FUTURE - Geozet Geozet -Wold Decorator Massive User Generated AR Contents Sharing Service Geozet All free, user generated AR content sharing Creating generated AR With Geozet client, users AR decorate real world Search targets for AR Users collect targets, iPhone/Android - select a target then add AR contents Watching targets with - add AR contents (thus an AR combo) - store it on Geozet server for later search (by zARTIS) - 3D files - Audio/Video Geozet Client Geozet & zARTIS - share it with friends answers - Texts/Images - detection results - current POIs - tracking information Unlimited number of channels per users zARTIS - AR contents - 15,000 targets per channel - either collected one at a time or all targets can be dumped at once - shared with friends (Facebook, Twitter and Geozet account) Hosting Premium Content Partner(PCP) Channels Users authors AR Geozet Server - Inviting premium contents providers for higher user repeat rate contents over the map - Potential partners : Large media, Consumer product manufacturers, - 3D files Restaurant Chains, Museums, Governments/NGO - Audio/Video - Texts/Images - POIs Geozet.com Unfair Advantages - ARGoo : Instant AR content creation - zARTIS : Massive # of Targets created & sharedCreating generated AR - Geozet API : Interactive AR contents are easily built Geozet Web - Geozet Billing : Revenue Generation for 3rd party - It’s All FREE!! 21
  22. 22. Zenitum, FUTURE - Geozet ario s Geozet -Wold Decorator e n 3D AR Contents on licensed products e ScUsag 22
  23. 23. Zenitum, FUTURE - Geozet ario s Geozet -Wold Decorator e n Massive AR contents on Supermarket aisles e ScUsag 23
  24. 24. Zenitum, FUTURE - Geozet ario s Geozet -Wold Decorator e n Merchandizes information & promotion on the fly e ScUsag 24
  25. 25. Zenitum, FUTURE - Geozet ario s Geozet -Wold Decorator e n Style & location guide e ScUsag 25
  26. 26. Zenitum, FUTURE - Products (from 2013) MMOMRG Platform Service Massive Multiplayers Online Mixed Reality Game Service MMOG MMOMRG Massive Multiplayers Online Games Massive Multiplayers Online Mixed Reality Gamesa multiplayer video game which is capable of supporting a multiplayer mixed reality game which is capable ofhundreds or thousands of players simultaneously. supporting thousands of players simultaneously. played on the Internet played both on the Internet and the Real world feature at least one persistent world feature multiple layers of persistent world on the Real world played on PC, consoles and mobile devices played on any devices with cameras for real world tracking Merging two worlds and Gaming everywhere 26
  27. 27. Zenitum, FUTURE - Products (from 2013) MMOMRG : project Nabokov plunge your self into Mixed Worlds Real world Tracking Gestural User InterfaceReal world Tracking Creates 3D coordinate Hands Tracking Gesture TrackingSceneTrack from Zenitum EgoMO from Zenitum Wearable Displays MMOG Platforms Project Nabokov • started July, 2010 (to replace D-Track engine) • SceneTrack ver. 1.0 released July, 2011 Ultra Light HMD Contact Lens Display • ARGoo + 20K object search/client will be released in Dec. 2011 MMOG Server Platform • Beta version of Nabokov launched in the early 2013HMDs from Brother, Vuzix, NTT DoCoMo (able to track up to 711 ft2 ,or 66 m2 indoor space first) MMOG Platform from existing 3D games 27
  28. 28. Zenitum, FUTURE - Products (from 2013) MMOMRG : project Nabokov escape & venture to the virtual worlds on earth Mixed Reality Games project Nabokov City wide virtual layers of game worlds Real spaces (i.e. rooms, buildings & cities) become game worlds Computer generated virtual worlds overlaid onto the real world The physical world becomes background frames for MR games MR designers replace real ads. with virtual ads. Shop owners can offers free/paid virtual goods within their shops Ordinary street billboards can be game targets for more engagement Once logged in, a user scans her room to extract 3D coordinates MRG system stores the coordinates as virtual worlds The user can download spatially matching games for free/paid worm holes between layers Mixed Reality #n : Design your own fantasy worlds Mixed Reality #3 : Dystopian future action games Mixed Reality #2 : Zombie Street with FPS games Mixed Reality #1 : Medieval role playing games Real & Physical World 28
  29. 29. Zenitum, FUTURE - Products (from 2013) MMOMRG : project Nabokov a huge eco-system for contents developers & users Purchasing Goggles/HMD the contents 3D Contents MMOMRG Platform Providers Nabokov Server Life-size 3D contents Freemium Developer API Game Developers Contents World MMO Nabokov Client Real & Physical Space Education Server Server Server Contents Tablets/Smart phones Adult Virtual Currency Platform Contents Real & Physical Space Virtual currency & more Virtual Currency to Real Currencies Life-size 3D contents Nabokov Interface to other payment services (i.e. PayPal) ClientContent Developers MMOMRG Platform MMOMRG Users (selecting game worlds) 29
  30. 30. Zenitum, FUTURE - New Strategy Addressing Massive Users Directly tapping into large users with right AR technologyB2C (Freemium) MMOMRG Platform - Nabokov Nabokov Developer API (zMART w/ SceneTrack) Geozet.com (DIY AR + Location based)B2B (Licensing) ARAD Platform (w/ ARGoo + zARTIS) zMART SDK (w/ zFT) zMART SDK (w/ ARGoo) zarCORE v. 1.0 (w/ ARGoo for chip makers) zarCORE v. 2.0 (w/ SceneTrack for chip makers) 2011 2012 2013 zFT ARGoo SceneTrack • Requires Pre-processing • No Pre-processing • Tracks real 3D space • Tracking small targets • Tracking small targets • Instant target generation (w/ ARGoo)AR Technologies • 5 targets/app • 5 targets/app zARTIS (v. 1.0) • Massive targets and Scene tracking zARTIS (v. 2.0) • Massive target processing (w/ zARTIS ver. 2.0) • Massive target processing • Real-time target recognition • Real-time target recognition • ~10,000 targets /channel • ~60,000 targets/channel zarCORE (v.1.0) zarCORE (v.2.0) • ARGoo based IP Core • SceneTrack based IP Core • AR features on microprocessors • AR features on microprocessors • ~100x better than SW only • 3D space tracking on chip level 2011 2012 2013 30
  31. 31. Zenitum, FUTURE - Target Markets MMOMRG Platform targets global mobile and online game market $44 Billion in 2014 50% of Global Game Market (est. $87 b.) MMOMRG Platform (project Nabokov) Light & Casual Gamers Heavy & Hardcore Gamers Professional Productivity Tools Educational Contents Object/Location based Knowledge Management Mobile commerce 3D Adult contents Tourism Religious Contents Immersive Interfaces 3D Advertisement & other unexpected uses... 31
  32. 32. Zenitum, FUTURE - Business Strategy Cross Platform Reinforcement Helping SW-HW support each other to create more values Providing based algorithm core Software Algorithm Hardware Algorithm SW Tracking Engine Engine IP Core New & Improved Algorithm Offering AR features on mobile devices New requirements & opportunities Cross platform AR Developer tools Application Processor Sales project Nabokov MMOMRG Platform New AR/MR applications run on the AR devices 3rd Party Developers New AR Devices zMART SDK OEM SDK More AR optimized devices to be used for the applications 32
  33. 33. Appendix Technology-Market Relation addressable market grow bigger with better solutions Mobile AR Games or Ads Mobile SNS or SNG AR object tracking & search AR Education (AR on books) AR User interface on devices AR on automobiles Nabokov Platform • up 100,000 targets/client • No pre-processing of targets zarCORE • 500 times faster tracking • practical AR based services NEW AR devicesMarket Size Mobile AR Games or Ads NEW entertainment Mobile SNS or SNG NEW education AR object tracking & search Life-size AR on streets AR Education (AR on books) Real space tracking Industrial applications Mobile AR Games or Ads • up 20,000 targets/client • No pre-processing of targets ARAD Platform (construction & factory sites) Mobile SNS or SNG • up 10 targets/app • practical AR based services zarCORE • up 500,000 targets/client • tracking the physical space • No pre-processing of targets • 2,500 times faster tracking • AR based SNS/SNG • life-long illusion of merging virtual & real in real-time Mobile AR Games or Ads - virtual lovers, friends & pets • up 10 targets/app • pre-processing is MUST ARAD Platform - immersive education • Simple but novelty app only Instant target generation & other NFT Technologies 2010 2011 2012 2013 33