This document discusses optical see-through head-mounted displays (OST-HMDs) in augmented reality. It covers the history of OST-HMDs and VR HMDs, challenges with OST-HMDs like field of view and latency, and approaches to improve realism through better visuals, spatial alignment, and temporal synchronization. The document also envisions future applications of OST-HMDs for vision augmentation and enhancement. These could involve analyzing individual eyes, compensating for visual impairments, and even predicting vision to enhance human capabilities.

1. Introduction to Optical See-
Through Head-Mounted Displays
in Augmented Reality
Yuta Itoh
23th May, 2016

2. ~2016.03: Ph.D. at TU Munich,
@Germany
2016.04~: Project Assistant Prof.
@Sugimoto lab
Yuta Itoh
Research topics:
OST-HMDs, Eye tracking, AH
Research: http://campar.in.tum.de/Main/YutaItoh

3. Augmented Reality (AR)
©IKEA, 2013

4. Augmented Reality (AR)
Feiner, Steven, Blair Macintyre, and Dorée
Seligmann. "Knowledge-based augmented
reality." Communications of the ACM 36.7
(1993): 53-62.
Hirota, Gentaro, et al. "Superior
augmented reality registration by
integrating landmark tracking and
magnetic tracking." ACM SIGGRAPH
1996.
Fuse virtual contents into the reality

5. Augmented Reality (AR)
[1] Klinker et al. ISMR’99, [2] Kato and Bilinghurst, [3] Kaufmann and Schmalstieg 2003
[3] IWAR’99, Sony Computer Entertainment, [4] Schwerdtfeger et al. VRST 2008, [5] Bichlmeier et al. ISMAR 20017

6. Augmented Reality (AR)
Research collections from FAR, TU Munich (http://campar.in.tum.de/Chair/ResearchAr)

7. Fuchs, Henry, et al. "Optimizing a head-tracked stereo display
system to guide hepatic tumor ablation." Studies in health
technology and informatics 132 (2007): 126-131.
Augmented Reality (AR)
Fuchs, Henry, et al. Augmented reality visualization for
laparoscopic surgery. Springer Berlin Heidelberg, 1998.

8. Non-Optical (Video) See-Through
Head-Mounted Displays (HMD)
Optical See-Through (OST-HMD)

9. VR
HMDs
Video
See-Thro.
Optical
See-Thro.
Opaque
display
Scene camera

10. Sight ©Robot Genius Film, 2012
https://vimeo.com/46304267

11. Hugo Gernsback,
TV Glasses
1963

12. Ivan Sutherland, The Sword of Damocles
1968

13. Ivan Sutherland, The Sword of Damocles
1968

14. 2000- (VR HMD)
Toshiba, 2006 Oculus Rift DK2, 2014
Photo from IFIXIT

15. 2013-

16. 2013-
Google Glass, 2013
Photo by Mikepanhu
Don't Be A Glasshole | Mashable

17. 2016 (VR HMDs)
Oculus Rift, 2016HTC Vive, 2016
Photo from Tom's Hardware

18. 2016
Meta 2, 2016 ODG R-7, 2016
EPSON BT300, 2016

19. Kotaku, Minecraft Hololens demo at E3 2015
https://www.youtube.com/watch?v=xgakdcEzVwg

20. 2016-
https://twitter.com/ksasao/status/715569511684911105
Microsoft, Hololens

21. 2016-
Magic Leap, 2016

22. ©Microsoft, 2015, 2016
How to make AR real?

23. Issues on OST-HMDs
Design Factors
Photo by Mikepanhu

24. Field of View
Issues on OST-HMD
https://twitter.com/ksasao/status/715569511684911105

25. Field of View
Issues on OST-HMD
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused Point Light Sources”Maimone et al., TOG’14

26. Issues on OST-HMD
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused Point Light Sources”Maimone et al., TOG’14
Field of View

27. Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
Temporal
[Zheng ’14] [Lincoln ’16]

28. Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
Temporal
[Zheng ’14] [Lincoln ’16]

29. Temporal Realism in OST-HMDs
“How fast is fast enough? : a study of the effects of latency in direct-
touch pointing tasks”Jota et al. CH’13
https://www.youtube.com/watch?v=PCbSTj7LjJg
“latencies down to 2.38 ms are required to alleviate
user perception when dragging”

30. Temporal Realism in OST-HMDs
Digital Light Processing Projector
“Minimizing Latency for Augmented Reality Displays: Frames
Considered Harmful”Zheng et al. ISMAR’14

31. Temporal Realism in OST-HMDs
Lincoln, Peter, et al. "From Motion to Photons in 80 Microseconds: Towards Minimal
Latency for Virtual and Augmented Reality." (2016), IEEE VR 2016 (Best paper award)

32. Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
Temporal
[Zheng ’14] [Lincoln ’16]

33. Visual Realism (including photorealism)
Occlusion, Depth, Color, etc..
Lee and woo, 2009
Kiyokawa et al. 2003
Liu et al. 2008

34. Occlusion (physical“Z-buffer”)
Kiyokawa et al. 2003

35. Depth (eye accommodation)
Liu et al. 2008
Konrad et al. 2016

36. Depth (eye accommodation)
The Light Field Stereoscope,
Huan et al., SIGGRPAH 2015

37. Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
Temporal
[Zheng ’14] [Lincoln ’16]

38. Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through-
kalibrierung fuer mobile augmented reality assistenzsysteme. Michael Schenk, Hrsg, 6(7).
Spatial Realism in OST-HMDs

39. Spatial Realism in OST-HMDs

40. Optics Designs of OST-HMDs
(Free-form)
prism
Point
source
Waveguide
Laser
source
Half-mirror
Eye
Prism-based
Waveguide-based
Half-mirror
Hainich and Bimber, Displays Fundamentals and Applications, 2011
Retinal Scanning

41. Eye-HMD Calibration
3D2D

42. Eye-HMD Calibration
3D

43. Off-axis Pinhole camera
3D2D

44. Manual method (De facto standard)
[Tuceryan ’00] [Genc ’02]
SPAAM
Single Point Active
Alignment Method

45. SPAAM (Manual Calibration) video

46. Track the position!
Itoh and Klinker, IEEE 3DUI 2014, IEEE ISMAR 2014
Automated method

47. OST-HMD with eye-tracking
Hua and Gao, SPIE, 2012

48. VR-HMD with eye-tracking
Stengel et al., ACM Multimedia, 2015

49. Better understandings of
HMD and Eye-ball optics
Locational Realism revisited

50. Eye
Environment
Medium
View-Dependent Lens Distortion
Distorted ray
HMD
optics
Image source
Itoh and Klinker IEEE TVCG‘15 (VR ’15) & IEEE ISMAR‘15

53. Light Field Calibration
World camera
User-view camera
Itoh and Klinker IEEE TVCG‘15 (VR ’15) & IEEE ISMAR‘15

54. Learned distortion maps
IEEE TVCG‘15 (VR ’15) & IEEE ISMAR‘15

55. View-Dependent Point-Spread Func.

56. Corneal-Imaging Calibration
Plopski et al., IEEE TVCG‘15 (VR ’15)

57. Schematic Eye for Virtual Environments
Jones et al., IEEE 3DUI 2016

58. What would happen in 20XX???
Sight ©Robot Genius Film, 2012

59. Vision Augmentation
Analysis of individual eyes
Computational photography
+

60. Low-vision devices
Peli, Eli. "Vision multiplexing: an engineering approach to vision rehabilitation device development." Optometry &
Vision Science 78.5 (2001): 304-315.
Peli, Eli. "Treating with spectacle lenses: a novel idea!?." Optometry & Vision Science 79.9 (2002): 569-580.

61. Enhancing Color
Toshiyuki Amano, Oliver Bimber, and Anselm Grundhöfer. "Appearance Enhancement for Visually Impaired with
Projector Camera Feedback.“, TECHNICAL REPORT, BAUHAUS-UNIVERSITY WEIMAR, JANUARY 2009

62. Broecker, Markus, Ross T. Smith, and Bruce H. Thomas. "Adaptive substrate for
enhanced spatial augmented reality contrast and resolution." transport 3.6 (2011): 22.
Enhancing Contrast

63. Huang, Fu-Chung, et al. "Computational light field display for correcting visual
aberrations." ACM SIGGRAPH 2013 Posters. ACM, 2013.
De-blurring displays

64. De-blurring OST-HMD screen
Oshima et al. ISMAR’15

65. De-blurring eye via OST-HMD
Itoh and Klinker, Augmented Human‘15

66. Compensation
layer
Blurred
Enhanced
Itoh and Klinker, Augmented Human‘15
Sharp

67. In Science Fiction, a precog sees his short-term future…
Next, Nicholas Cage, TM & © Paramount (2007)

68. Vision Predication
Itoh et al., Augmented Human‘16

69. A proof-of-concept system
Tracking system
User-view
Camera
OST-HMD
Virtual
Image plane

70. Tracking system
User-view
Camera
OST-HMD
Virtual
Image plane
A proof-of-concept system

71.  Realism in OST-HMDs
 Better understandings of
Tracking, Graphics, HMD
optics, Eye balls, etc.
Summary
 Vision Augmentation
 Extend human capability
Questions & comments: yuta.itoh@keio.jp
Research: http://campar.in.tum.de/Main/YutaItoh

72. Resources related to OST-HMD&AR
“Optical See-Through AR Calibration: Methods for Current & Next
Generation Head-Mounted Displays”, Kenneth Moser
https://sites.google.com/site/krm104/project-updates/ieeevr2016tutorialslidesandmaterials
“How to calibrate your Optical See-Through Head Mounted Displays”,
Jens Grubert and Yuta Itohenneth Moser
http://www.slideshare.net/JensGrubert/google-glass-the-meta-and-co-how-to-calibrate-your-optical-
seethrough-head-mounted-displays
“2016 AR Summer School ”, Mark Billinghurst,
http://www.slideshare.net/marknb00/2016-ar-summer-school-lecture1