Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Head Mounted Displays: How to realize ultimate AR experiences?

2,270 views

Published on

Slides from a talk I gave at:
Wissens-Austausch-Workshop: The 3rd Workshop on Visualisierung großer Datenmengen in der Wissenschaft (VisDa3? WAW), DLR@Oberpfaffenhofen, Germany, Jun. 9 - 10, 2015.

For more detail of my work, see:
http://campar.in.tum.de/Main/YutaItoh

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Head Mounted Displays: How to realize ultimate AR experiences?

  1. 1. FAR: Fachgebiet Augmented Reality Technische Universität München Head Mounted Displays How to realize ultimate AR experiences? Yuta Itoh (TUM) Workshop Visualisierung großer Datenmengen, 9th June 2015 1
  2. 2. FAR: Fachgebiet Augmented Reality Prof. Gudrun Klinker 2/52
  3. 3. Industrial Augmented Reality 3/52
  4. 4. Head-Mounted Displays (HMDs) Optical See-Through (OST-HMD) Non Optical See-Through 4/52
  5. 5. OST-HMD in science fiction From Ironman, 2008 ©Marvel Entertainment 5/52
  6. 6. Everything is perfect in Science Fiction  World sensing  Image recognition  Eye tracking 6/52
  7. 7. 7/52
  8. 8. Arianespace, 2009Le Voyage dans la Lune, 1902 1902 2009 8/52
  9. 9. What is the current status of the OST-HMD technology? What’s still missing? 9/52
  10. 10. Hugo Gernsback, TV Glasses 1963 10/52
  11. 11. Ivan Sutherland, The Sword of Damocles 1968 11/52
  12. 12. 2013- 12/52
  13. 13. 13/52
  14. 14. 14/52
  15. 15. OST-HMDs Overwrite the reality NVIS nVisor ST60 15/52
  16. 16. Related Work | AR X-ray vision Avery, Ben, Christian Sandor, and Bruce H. Thomas. "Improving spatial perception for augmented reality x-ray vision." Virtual Reality Conference, 2009. VR 2009. IEEE. IEEE, 2009. 16/52
  17. 17. Related Work | Visual Assistance Hwang, Alex D., and Eli Peli. "An augmented-reality edge enhancement application for Google Glass." Optometry & Vision Science 91.8 (2014): 1021-1030. 17/52
  18. 18. Issues in OST-HMDs 18/52
  19. 19. Design Factor (non-OST) Toshiba, 2006 Oculus Rift DK2, 2014 Photo from IFIXIT 19/52
  20. 20. Design Factor Google Glass, 2013 Photo by Mikepanhu 20/52
  21. 21. Wide Field of View “Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses using Defocused Point Light Sources” Maimone et al., TOG’14 21/52
  22. 22. Wide Field of View Issues on OST-HMD “Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses using Defocused Point Light Sources” Maimone et al., TOG’14 22/52
  23. 23. Consistency issues in OST-HMDs Temporal [Zheng ’14] [Didier ’05] Visual [Kiyokawa ’01] [Liu ’08] [Lee ’09] Spatial [Azuma ’95] [Tuceryan ’00] [Genc ’02] Social Image from Google Glass: Don't Be A Glasshole | Mashable 23/52
  24. 24. Temporal Inconsist. in OST-HMD “How fast is fast enough? : a study of the effects of latency in direct-touch pointing tasks” Jota et al. CH’13 “latencies down to 2.38 ms are required to alleviate user perception when dragging” 24/52
  25. 25. Temporal Inconsist. in OST-HMD Digital Light Processing Projector “Minimizing Latency for Augmented Reality Displays: Frames Considered Harmful” Zheng et al. ISMAR’14 25/52
  26. 26. Visual Consistency Occlusion, Depth, Color, Shadow, etc.. Lee and woo, 2009 Kiyokawa et al. 2003 Liu et al. 2008 26/52
  27. 27. Occlusion (physical “Z-buffer”) 27/52
  28. 28. Depth (eye accommodation) Liu et al. 2008 28/52
  29. 29. Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through- kalibrierung fuer mobile augmented reality assistenzsysteme. Michael Schenk, Hrsg, 6(7). Spatial Calibration of OST-HMDs 29/52
  30. 30. Spatial Calibration of OST-HMDs 30/52
  31. 31. Eye-HMD Calibration 3D 2D 31/52
  32. 32. P is a Black Box Find 3D-2D Projection: 3D2D 32/52
  33. 33. 2D Say, is a Perspective Projection 33/52
  34. 34. Find 3D-2D Projection: In the Eye of the Beholder… = K*[R t] 34/52
  35. 35. Manual alignment – Intensive user interaction – User-dependent noise Predictive tracking for augmented reality RT Azuma - 1995 - ronaldazuma.com 35/52
  36. 36. Manual alignment SPAAM: Single Point Active Alignment Method – Medium user interaction – User-dependent noise Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR M Tuceryan, N Navab - Augmented Reality, 2000.(ISAR 2000). …, 2000 - ieeexplore.ieee.org Practical solutions for calibration of optical see-through devices Y Genc, M Tuceryan, N Navab - … of the 1st International Symposium on …, 2002 - dl.acm.org N times 36/52
  37. 37. SPAAM (Manual Calibration) video 37/52
  38. 38. Idea: user-perspective rendering Spindler, Martin, Wolfgang Büschel, and Raimund Dachselt. "Use your head: tangible windows for 3D information spaces in a tabletop environment."Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces. ACM, 2012. 38/52
  39. 39. Utilizes 3D Eye Localization –Interaction-free, thus do not bother users –More accurate than a realistic SPAAM setup Automated alignement Yuta Itoh, and Gudrun Klinker. "Interaction-free calibration for optical see-through head-mounted displays based on 3d eye localization." 3D User Interfaces (3DUI), 2014 IEEE Symposium on. IEEE, 2014. 39/52
  40. 40. 1. Estimate a 2D iris ellipse – Iris detector + Fitting by RANSAC 2. Back project it to 3D circle Robust real-time pupil tracking in highly off-axis images L Świrski, A Bulling, N Dodgson - … of the Symposium on Eye Tracking …, 2012 - dl.acm.org 3D Eye Position Estimation Image-based Eye Pose and Reflection Analysis for Advanced Interaction Techniques andScene Understanding C Nitschke, A. Nakazawa, H. Takemura - CVIM,[ …, 2011 - ci.nii.ac.jp 40/52
  41. 41. Estimated 3D Eye Position [m] >10 cm 41/52
  42. 42. Wearable eye-trackers Pupil Labs Tobii 42/52
  43. 43. OST-HMD + eye trackers? 43/52
  44. 44. OST-HMD + eye trackers? A compact, eye-tracked optical see-through head-mounted display H Hua, C Gao - Proc. SPIE, 2012 - ebooks.spiedigitallibrary.org 44/52
  45. 45. Pupil Center Cornea Reflection Villanueva ‘08 Cornea reflection 45
  46. 46. HMD Screen resembles a plane HMD Image Screen 46
  47. 47. Corneal Reflection 47
  48. 48. Detected Checkerboard in 2D Detected grid corners 48
  49. 49. Scene Reflection in the Eye 49
  50. 50. Detected and Projected Corners Reprojected Points (3D  2D) Original Image 50
  51. 51. Estimated Cornea Position Limbus tracking Corneal Imaging Plopski, Alexander, et al. "Corneal-imaging calibration for optical see-through head-mounted displays." Visualization and Computer Graphics, IEEE Transactions on 21.4 (2015): 481-490. 51
  52. 52. Optical See-Through HMDs are hard fun • Consistency issues (Spatial, visual, temporal, etc.) • Eye sensing will be a hot topic • Big players => big markets Takeaways: 52/52
  53. 53. Appendix 53/52
  54. 54. An ideal eye-HMD system Eye Image source Environment Combiner (a half mirror) 54/52
  55. 55. HMD-dependent parameters User-dependent parameters Image by Soerfm Photo by Laitr Keiows 55
  56. 56. • Shipments of smart watches&glasses –130 million by 2018 •10 times than 2013 Dawn of a Commercial Wearable Devices 56/52
  57. 57. Innovega iOptik 57/52
  58. 58. Simple No user interaction Accurate  better than Degraded manual calibrations Summary of INDICA Calibration of OST-HMDs using 3D eye position 58/52
  59. 59. Eye Position Estimation E < C1 Eyeball sphere C3 Eyeball orbit g2 g3 g1 C2 59

×