Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Introduction to Optical See-
Through Head-Mounted Displays
in Augmented Reality
Yuta Itoh
23th May, 2016
~2016.03: Ph.D. at TU Munich,
@Germany
2016.04~: Project Assistant Prof.
@Sugimoto lab
Yuta Itoh
Research topics:
OST-HMDs...
Augmented Reality (AR)
©IKEA, 2013
Augmented Reality (AR)
Feiner, Steven, Blair Macintyre, and Dorée
Seligmann. "Knowledge-based augmented
reality." Communic...
Augmented Reality (AR)
[1] Klinker et al. ISMR’99, [2] Kato and Bilinghurst, [3] Kaufmann and Schmalstieg 2003
[3] IWAR’99...
Augmented Reality (AR)
Research collections from FAR, TU Munich (http://campar.in.tum.de/Chair/ResearchAr)
Fuchs, Henry, et al. "Optimizing a head-tracked stereo display
system to guide hepatic tumor ablation." Studies in health
...
Non-Optical (Video) See-Through
Head-Mounted Displays (HMD)
Optical See-Through (OST-HMD)
VR
HMDs
Video
See-Thro.
Optical
See-Thro.
Opaque
display
Scene camera
Sight ©Robot Genius Film, 2012
https://vimeo.com/46304267
Hugo Gernsback,
TV Glasses
1963
Ivan Sutherland, The Sword of Damocles
1968
Ivan Sutherland, The Sword of Damocles
1968
2000- (VR HMD)
Toshiba, 2006 Oculus Rift DK2, 2014
Photo from IFIXIT
2013-
2013-
Google Glass, 2013
Photo by Mikepanhu
Don't Be A Glasshole | Mashable
2016 (VR HMDs)
Oculus Rift, 2016HTC Vive, 2016
Photo from Tom's Hardware
2016
Meta 2, 2016 ODG R-7, 2016
EPSON BT300, 2016
Kotaku, Minecraft Hololens demo at E3 2015
https://www.youtube.com/watch?v=xgakdcEzVwg
2016-
https://twitter.com/ksasao/status/715569511684911105
Microsoft, Hololens
2016-
Magic Leap, 2016
©Microsoft, 2015, 2016
How to make AR real?
Issues on OST-HMDs
Design Factors
Photo by Mikepanhu
Field of View
Issues on OST-HMD
https://twitter.com/ksasao/status/715569511684911105
Field of View
Issues on OST-HMD
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused Point ...
Issues on OST-HMD
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused Point Light Sources”...
Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Gru...
Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Gru...
Temporal Realism in OST-HMDs
“How fast is fast enough? : a study of the effects of latency in direct-
touch pointing tasks...
Temporal Realism in OST-HMDs
Digital Light Processing Projector
“Minimizing Latency for Augmented Reality Displays: Frames...
Temporal Realism in OST-HMDs
Lincoln, Peter, et al. "From Motion to Photons in 80 Microseconds: Towards Minimal
Latency fo...
Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Gru...
Visual Realism (including photorealism)
Occlusion, Depth, Color, etc..
Lee and woo, 2009
Kiyokawa et al. 2003
Liu et al. 2...
Occlusion (physical“Z-buffer”)
Kiyokawa et al. 2003
Depth (eye accommodation)
Liu et al. 2008
Konrad et al. 2016
Depth (eye accommodation)
The Light Field Stereoscope,
Huan et al., SIGGRPAH 2015
Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Gru...
Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through-
kalibrierung fuer mobile augm...
Spatial Realism in OST-HMDs
Optics Designs of OST-HMDs
(Free-form)
prism
Point
source
Waveguide
Laser
source
Half-mirror
Eye
Prism-based
Waveguide-bas...
Eye-HMD Calibration
3D2D
Eye-HMD Calibration
3D
Off-axis Pinhole camera
3D2D
Manual method (De facto standard)
[Tuceryan ’00] [Genc ’02]
SPAAM
Single Point Active
Alignment Method
SPAAM (Manual Calibration) video
Track the position!
Itoh and Klinker, IEEE 3DUI 2014, IEEE ISMAR 2014
Automated method
OST-HMD with eye-tracking
Hua and Gao, SPIE, 2012
VR-HMD with eye-tracking
Stengel et al., ACM Multimedia, 2015
Better understandings of
HMD and Eye-ball optics
Locational Realism revisited
Eye
Environment
Medium
View-Dependent Lens Distortion
Distorted ray
HMD
optics
Image source
Itoh and Klinker IEEE TVCG‘15 ...
Light Field Calibration
World camera
User-view camera
Itoh and Klinker IEEE TVCG‘15 (VR ’15) & IEEE ISMAR‘15
Learned distortion maps
IEEE TVCG‘15 (VR ’15) & IEEE ISMAR‘15
View-Dependent Point-Spread Func.
Corneal-Imaging Calibration
Plopski et al., IEEE TVCG‘15 (VR ’15)
Schematic Eye for Virtual Environments
Jones et al., IEEE 3DUI 2016
What would happen in 20XX???
Sight ©Robot Genius Film, 2012
Vision Augmentation
Analysis of individual eyes
Computational photography
+
Low-vision devices
Peli, Eli. "Vision multiplexing: an engineering approach to vision rehabilitation device development." ...
Enhancing Color
Toshiyuki Amano, Oliver Bimber, and Anselm Grundhöfer. "Appearance Enhancement for Visually Impaired with
...
Broecker, Markus, Ross T. Smith, and Bruce H. Thomas. "Adaptive substrate for
enhanced spatial augmented reality contrast ...
Huang, Fu-Chung, et al. "Computational light field display for correcting visual
aberrations." ACM SIGGRAPH 2013 Posters. ...
De-blurring OST-HMD screen
Oshima et al. ISMAR’15
De-blurring eye via OST-HMD
Itoh and Klinker, Augmented Human‘15
Compensation
layer
Blurred
Enhanced
Itoh and Klinker, Augmented Human‘15
Sharp
In Science Fiction, a precog sees his short-term future…
Next, Nicholas Cage, TM & © Paramount (2007)
Vision Predication
Itoh et al., Augmented Human‘16
A proof-of-concept system
Tracking system
User-view
Camera
OST-HMD
Virtual
Image plane
Tracking system
User-view
Camera
OST-HMD
Virtual
Image plane
A proof-of-concept system
 Realism in OST-HMDs
 Better understandings of
Tracking, Graphics, HMD
optics, Eye balls, etc.
Summary
 Vision Augmenta...
Resources related to OST-HMD&AR
“Optical See-Through AR Calibration: Methods for Current & Next
Generation Head-Mounted Di...
Introduction to Optical See-Through HMDs in AR
Introduction to Optical See-Through HMDs in AR
Upcoming SlideShare
Loading in …5
×

Introduction to Optical See-Through HMDs in AR

3,237 views

Published on

This is a modified version of lecture notes originally talked in an AR lecture at Keio University on 23th May, 2016.

Published in: Technology
  • Be the first to comment

Introduction to Optical See-Through HMDs in AR

  1. 1. Introduction to Optical See- Through Head-Mounted Displays in Augmented Reality Yuta Itoh 23th May, 2016
  2. 2. ~2016.03: Ph.D. at TU Munich, @Germany 2016.04~: Project Assistant Prof. @Sugimoto lab Yuta Itoh Research topics: OST-HMDs, Eye tracking, AH Research: http://campar.in.tum.de/Main/YutaItoh
  3. 3. Augmented Reality (AR) ©IKEA, 2013
  4. 4. Augmented Reality (AR) Feiner, Steven, Blair Macintyre, and Dorée Seligmann. "Knowledge-based augmented reality." Communications of the ACM 36.7 (1993): 53-62. Hirota, Gentaro, et al. "Superior augmented reality registration by integrating landmark tracking and magnetic tracking." ACM SIGGRAPH 1996. Fuse virtual contents into the reality
  5. 5. Augmented Reality (AR) [1] Klinker et al. ISMR’99, [2] Kato and Bilinghurst, [3] Kaufmann and Schmalstieg 2003 [3] IWAR’99, Sony Computer Entertainment, [4] Schwerdtfeger et al. VRST 2008, [5] Bichlmeier et al. ISMAR 20017
  6. 6. Augmented Reality (AR) Research collections from FAR, TU Munich (http://campar.in.tum.de/Chair/ResearchAr)
  7. 7. Fuchs, Henry, et al. "Optimizing a head-tracked stereo display system to guide hepatic tumor ablation." Studies in health technology and informatics 132 (2007): 126-131. Augmented Reality (AR) Fuchs, Henry, et al. Augmented reality visualization for laparoscopic surgery. Springer Berlin Heidelberg, 1998.
  8. 8. Non-Optical (Video) See-Through Head-Mounted Displays (HMD) Optical See-Through (OST-HMD)
  9. 9. VR HMDs Video See-Thro. Optical See-Thro. Opaque display Scene camera
  10. 10. Sight ©Robot Genius Film, 2012 https://vimeo.com/46304267
  11. 11. Hugo Gernsback, TV Glasses 1963
  12. 12. Ivan Sutherland, The Sword of Damocles 1968
  13. 13. Ivan Sutherland, The Sword of Damocles 1968
  14. 14. 2000- (VR HMD) Toshiba, 2006 Oculus Rift DK2, 2014 Photo from IFIXIT
  15. 15. 2013-
  16. 16. 2013- Google Glass, 2013 Photo by Mikepanhu Don't Be A Glasshole | Mashable
  17. 17. 2016 (VR HMDs) Oculus Rift, 2016HTC Vive, 2016 Photo from Tom's Hardware
  18. 18. 2016 Meta 2, 2016 ODG R-7, 2016 EPSON BT300, 2016
  19. 19. Kotaku, Minecraft Hololens demo at E3 2015 https://www.youtube.com/watch?v=xgakdcEzVwg
  20. 20. 2016- https://twitter.com/ksasao/status/715569511684911105 Microsoft, Hololens
  21. 21. 2016- Magic Leap, 2016
  22. 22. ©Microsoft, 2015, 2016 How to make AR real?
  23. 23. Issues on OST-HMDs Design Factors Photo by Mikepanhu
  24. 24. Field of View Issues on OST-HMD https://twitter.com/ksasao/status/715569511684911105
  25. 25. Field of View Issues on OST-HMD “Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses using Defocused Point Light Sources”Maimone et al., TOG’14
  26. 26. Issues on OST-HMD “Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses using Defocused Point Light Sources”Maimone et al., TOG’14 Field of View
  27. 27. Realism in AR (with OST HMDs) Visual [Kiyokawa ’01] [Liu ’08] [Lee ’09] Spatial [Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08] Temporal [Zheng ’14] [Lincoln ’16]
  28. 28. Realism in AR (with OST HMDs) Visual [Kiyokawa ’01] [Liu ’08] [Lee ’09] Spatial [Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08] Temporal [Zheng ’14] [Lincoln ’16]
  29. 29. Temporal Realism in OST-HMDs “How fast is fast enough? : a study of the effects of latency in direct- touch pointing tasks”Jota et al. CH’13 https://www.youtube.com/watch?v=PCbSTj7LjJg “latencies down to 2.38 ms are required to alleviate user perception when dragging”
  30. 30. Temporal Realism in OST-HMDs Digital Light Processing Projector “Minimizing Latency for Augmented Reality Displays: Frames Considered Harmful”Zheng et al. ISMAR’14
  31. 31. Temporal Realism in OST-HMDs Lincoln, Peter, et al. "From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality." (2016), IEEE VR 2016 (Best paper award)
  32. 32. Realism in AR (with OST HMDs) Visual [Kiyokawa ’01] [Liu ’08] [Lee ’09] Spatial [Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08] Temporal [Zheng ’14] [Lincoln ’16]
  33. 33. Visual Realism (including photorealism) Occlusion, Depth, Color, etc.. Lee and woo, 2009 Kiyokawa et al. 2003 Liu et al. 2008
  34. 34. Occlusion (physical“Z-buffer”) Kiyokawa et al. 2003
  35. 35. Depth (eye accommodation) Liu et al. 2008 Konrad et al. 2016
  36. 36. Depth (eye accommodation) The Light Field Stereoscope, Huan et al., SIGGRPAH 2015
  37. 37. Realism in AR (with OST HMDs) Visual [Kiyokawa ’01] [Liu ’08] [Lee ’09] Spatial [Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08] Temporal [Zheng ’14] [Lincoln ’16]
  38. 38. Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through- kalibrierung fuer mobile augmented reality assistenzsysteme. Michael Schenk, Hrsg, 6(7). Spatial Realism in OST-HMDs
  39. 39. Spatial Realism in OST-HMDs
  40. 40. Optics Designs of OST-HMDs (Free-form) prism Point source Waveguide Laser source Half-mirror Eye Prism-based Waveguide-based Half-mirror Hainich and Bimber, Displays Fundamentals and Applications, 2011 Retinal Scanning
  41. 41. Eye-HMD Calibration 3D2D
  42. 42. Eye-HMD Calibration 3D
  43. 43. Off-axis Pinhole camera 3D2D
  44. 44. Manual method (De facto standard) [Tuceryan ’00] [Genc ’02] SPAAM Single Point Active Alignment Method
  45. 45. SPAAM (Manual Calibration) video
  46. 46. Track the position! Itoh and Klinker, IEEE 3DUI 2014, IEEE ISMAR 2014 Automated method
  47. 47. OST-HMD with eye-tracking Hua and Gao, SPIE, 2012
  48. 48. VR-HMD with eye-tracking Stengel et al., ACM Multimedia, 2015
  49. 49. Better understandings of HMD and Eye-ball optics Locational Realism revisited
  50. 50. Eye Environment Medium View-Dependent Lens Distortion Distorted ray HMD optics Image source Itoh and Klinker IEEE TVCG‘15 (VR ’15) & IEEE ISMAR‘15
  51. 51. Light Field Calibration World camera User-view camera Itoh and Klinker IEEE TVCG‘15 (VR ’15) & IEEE ISMAR‘15
  52. 52. Learned distortion maps IEEE TVCG‘15 (VR ’15) & IEEE ISMAR‘15
  53. 53. View-Dependent Point-Spread Func.
  54. 54. Corneal-Imaging Calibration Plopski et al., IEEE TVCG‘15 (VR ’15)
  55. 55. Schematic Eye for Virtual Environments Jones et al., IEEE 3DUI 2016
  56. 56. What would happen in 20XX??? Sight ©Robot Genius Film, 2012
  57. 57. Vision Augmentation Analysis of individual eyes Computational photography +
  58. 58. Low-vision devices Peli, Eli. "Vision multiplexing: an engineering approach to vision rehabilitation device development." Optometry & Vision Science 78.5 (2001): 304-315. Peli, Eli. "Treating with spectacle lenses: a novel idea!?." Optometry & Vision Science 79.9 (2002): 569-580.
  59. 59. Enhancing Color Toshiyuki Amano, Oliver Bimber, and Anselm Grundhöfer. "Appearance Enhancement for Visually Impaired with Projector Camera Feedback.“, TECHNICAL REPORT, BAUHAUS-UNIVERSITY WEIMAR, JANUARY 2009
  60. 60. Broecker, Markus, Ross T. Smith, and Bruce H. Thomas. "Adaptive substrate for enhanced spatial augmented reality contrast and resolution." transport 3.6 (2011): 22. Enhancing Contrast
  61. 61. Huang, Fu-Chung, et al. "Computational light field display for correcting visual aberrations." ACM SIGGRAPH 2013 Posters. ACM, 2013. De-blurring displays
  62. 62. De-blurring OST-HMD screen Oshima et al. ISMAR’15
  63. 63. De-blurring eye via OST-HMD Itoh and Klinker, Augmented Human‘15
  64. 64. Compensation layer Blurred Enhanced Itoh and Klinker, Augmented Human‘15 Sharp
  65. 65. In Science Fiction, a precog sees his short-term future… Next, Nicholas Cage, TM & © Paramount (2007)
  66. 66. Vision Predication Itoh et al., Augmented Human‘16
  67. 67. A proof-of-concept system Tracking system User-view Camera OST-HMD Virtual Image plane
  68. 68. Tracking system User-view Camera OST-HMD Virtual Image plane A proof-of-concept system
  69. 69.  Realism in OST-HMDs  Better understandings of Tracking, Graphics, HMD optics, Eye balls, etc. Summary  Vision Augmentation  Extend human capability Questions & comments: yuta.itoh@keio.jp Research: http://campar.in.tum.de/Main/YutaItoh
  70. 70. Resources related to OST-HMD&AR “Optical See-Through AR Calibration: Methods for Current & Next Generation Head-Mounted Displays”, Kenneth Moser https://sites.google.com/site/krm104/project-updates/ieeevr2016tutorialslidesandmaterials “How to calibrate your Optical See-Through Head Mounted Displays”, Jens Grubert and Yuta Itohenneth Moser http://www.slideshare.net/JensGrubert/google-glass-the-meta-and-co-how-to-calibrate-your-optical- seethrough-head-mounted-displays “2016 AR Summer School ”, Mark Billinghurst, http://www.slideshare.net/marknb00/2016-ar-summer-school-lecture1

×