Keynote Virtual Efficiency Congress 2012


Published on

My keynote at the Virtual Efficiency Congress 2012 ( )
More information:

Published in: Technology, Education
  • Be the first to comment

Keynote Virtual Efficiency Congress 2012

  1. 1. Taking Augmented Reality out of the Laboratory and into the Real WorldDr Christian SandorDirector: Magic Vision LabSenior Lecturer:School of Computer and Information ScienceUniversity of South Australia
  2. 2. Stuttgart University TU Munich 2012 1975-2005 Canon 2005-2007ColumbiaUniversity 2004 University of South Australia since 2008
  3. 3. 2009 2010 2011Students Research Sponsors2 PhD [2013: +3; -1] Augmented Reality (AR)2 Master Human-Computer Interaction4 Internship Haptics Visualization
  4. 4. AR ExampleDemo: Tracker by Gerhard Reitmayr (TU Graz, Austria)
  5. 5. Augmented Reality [Azuma 1997] 1. Combines real and virtual 2. Interactive in realtime 3. Registered in 3–D [Milgram & Kishino, 1994]2 Challenges in Developing User Interfaces for Ubiquitous Augmented Reality Mixed Reality (MR) Real Augmented Augmented Virtual Environment Reality (AR) Virtuality (AV) Environment
  6. 6. AR in the Real World1584: Pepper’s Ghost[Giambattista della Porta]Early 1990s: Boeing coinsterm AR for their wireassembly application2002: Intelligent WeldingGun [Klinker & BMW]
  7. 7. Recent AR Trends
  8. 8. Current State of AR: low-level = solved!Essential technology: tracking(where is the camera in the real world?) Parallel Tracking and Mapping for Small AR Workspaces [Klein & Murray, 2007] KinectFusion [Newcombe et al., 2011]
  9. 9. Current State of AR:Challenge = High-Level 1. Applications Industrial Design (with Canon) AR Browser (with Nokia, Samsung, Nvidia) Medical Games Other industrial applications (training, maintenance, planning, ...) ... 2. Human-Computer Interaction Human Perception of AR Usability Providing more versatile AR interfaces
  10. 10. Our Approach Augmented RealityVisualization:“seeing the unseen”[McCormick, 1988]Haptics: AR for thesense of touch Visualization Haptics
  11. 11. AR & VisualizationAR & Haptics
  12. 12. Motivation Problems with most AR browsers: Pieces of isolated information instead of one integrated visualization Bad ergonomics Small screen problem becomes even worse Extremely limited visualizations Challenging: occlusions, small field of view C D T E O D B T T T A O Dfield of view field of view view frustrum user viewpoint user viewpoint
  13. 13. Naive Overlay Benjamin Avery, Bruce H. Thomas, Wayne Piekarski. ISMAR 2008.
  14. 14. Edge-based X-Ray Ben Avery, Christian Sandor, Bruce Thomas. VR 2009.
  15. 15. AR X-Ray Vision:Examples
  16. 16. Edge-based X-Ray:Limitations
  17. 17. Saliency X-Ray Sandor et al. ISMAR 2010
  18. 18. Concept: Saliency-based X-RayScene Foreground Scene Background 1. Saliency Map Computation Salient Foreground Salient Background 2, Composition Salient Salient reground Fo Bac kground
  19. 19. ly on visual data. r g chnique uses saliency maps Mrg = f the occluder and occludedWe first compute the saliency Saliency Map max(r, g, b) Oc b min(r, g) ons. Second, we perform a lient regions in the occluder Computation g, b) M = max(r, by Figure 4: Saliency map computation: an input image is split into fe e occluded region are made ture maps which are across-scale subtracted to a single map Mc . These mapsPrevious and Mby are combined into mimic the recepti Mrg transparent. When salientInput Image Image fields of the human eye. The features mapsthe luminosity chann Motion is defined as observed changes in are combined to yie ding to their strength. With Figure 4: Saliency map computation: an input image is split into fea- the final saliencywhich are across-scale subtracted to mimic the receptive over time. maps map. ture hile still maintaining strong and fields inthe human eye. The addition and across-scale subtractio denote across scale features maps are combined to yield Contrasts of the dyadic feature pyramids are modeled as acro or the two stages (Saliency the final saliency map. scale subtraction Map across scale addition and across-scale subtraction.levels of t Red/Green Blue/Yellow denote between fine and coarse scaled and Motionomposition (Section 3.2), we Luminosity Map Opponency Map Opponency Map pyramid. For each of the features, a set of feature maps are geneX-ray technique. S 2 {3, 4}.2Features maps arecombined using using across-scale additi ated as: S {3, 4}. Features maps are combined across-scale addition Figure 5 ries of fi m, which only used edges as σ=0 to yield to yield conspicuity maps: = Pp Ps conspicuity maps: Ff ,p,s , , an ionally employs luminosity, 4 p+4 values. es the effect for luminosity, ⊖ where ⊖ represents the visual4feature f 2 {l, c, m}. p and s ref f = M p+4 M M ⊖ areC appliedM p 2 {2, 3, 4}, s = p + S, fin Fp,s σ=1 ⊖ are preserved. Figure 3(a,b) to pyramid levels and C = p=2 s=p+3as Fp,s This an σ=2 and occ σ=3 p=2 s=p+3 Finally, all conspicuity maps are combined to form the saliency with em σ=… map: 1 4 E VA Finally, all conspicuity Smaps  Ck = 3 k2{l,c,t} combined to form the salien are We hav ⊕map: saliency target ac At this point, a saliency map1 been created for an image, com- has  bining the hue, luminosity= motion features. In the next stage, gate vis S and Ck techniqu Saliency Map 3 k2{l,c,t} occluded and occluder regions are composed using their saliency Our d information to create the final AR X-ray image. capabili how it At this3.2 Composition map has been created for an image, com point, a saliency
  20. 20. Composition Occluder Io Occluded IdSource Images So SdSaliency Maps ⊖ Edge map E Combined Mask M Saliency Map ⊗ Occluder Io So Occluded Id Mask M ⊗ ⊗ ⊕ Final Composition Ic
  21. 21. C T B D A D melt volumeMelting Sandor et al. ISMAR 2009, VR 2010
  22. 22. More Melting Examples
  23. 23. Space-Distorting Visualizations Radial Distortion field of view after distortion C T ED B T C T DT D T B D A D A Dfield of view orginal view frustrum field of view user viewpoint Reconstructed Model Projected Video Image POI2 POI1 Ray Visualization
  24. 24. Space-Distorting Visualizations Radial Distortion
  25. 25. AR & VisualizationAR & Haptics
  26. 26. Our ISMAR 2011 Best Demo AwardCollaboration with Gerhard Reitmayr (TU Graz, Computer Vision) Matt Swoboda (Sony London, Computer Graphics)We won against 40 other demos Top labs: INRIA, Georgia Tech, TU Graz Top companies: Volkswagen, Sony, Nokia...
  27. 27. Current EvaluationAt ISMAR, we got very unexpectedfeedback from users: 20% reported a heat sensation 5% reported smelling fireNow: formal evaluation to validate thiseffect
  28. 28. Realtime Raymarching for Mobile Augmented Reality Graeme Jarvis⇤ Christian Sandor† Sean White‡ Magic Vision Lab Magic Vision Lab Nokia Research Center University of South Australia University of South Australia Nokia (a) (b)Figure 1: Raymarching on mobile phones enables us to display several effects on virtual objects that incorporate imagery from the physicalworld: from simple refractions (a), to dynamic, complex scenes (c). All renderings are realtime on an iPhone 4s.A BSTRACT (iPhone), Ray Tracer and Raytracing on Android, and many others but all render at multiple-seconds-per-frame.Augmented Reality (AR) is a technology that adds virtual visualinformation to the user’s view of the real world. Mobile Aug- Visual effects such as ambient occlusion, reflection, refraction,mented Reality (MAR) systems allow the user to take these aug- and dynamic soft shadow (as exampled in Figure 1) are process-mentations with them on their travels, building the foundation of a ing intensive and difficult to simulate using other visual algorithms,
  29. 29. Glass Sphere with Refractions
  30. 30. Conclusions: Scientific ContributionWe have co-pioneered: Augmented RealityAR & VisualizationWhite, Feiner (Columbia University)Kalkofen, Schmalstieg (TU Graz)AR & HapticsKnörlein, Harders (ETH Zurich) Visualization Haptics
  31. 31. Conclusions: Future Work Augmented RealityVisualization & HapticsUnder-explored! Visualization Haptics
  32. 32. Conclusions: Future Work Augmented RealityVisualization & HapticsUnder-explored!Full combination:Startrek’s Holodeck Visualization Haptics
  33. 33. More
  34. 34. Key Points For TakingAR into the Real WorldTracking: practically solvedChallenges: Applications AR Browser (with Nokia, Samsung, Nvidia) Industrial Design (with Canon) Medical Games Other industrial applications (training, maintenance, planning, ...) ... Human-Computer Interaction Human Perception of AR Usability Providing more versatile AR interfaces Thank You!