1. Ramesh Raskar Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA New Directions in Augmented Reality

6. Spatially Augmented Reality (SAR) HMD-VR Spatially Immersive-VR AR using HMD Spatially Augmented Reality e.g. CAVE Video or Optical see-through SAR, Shaderlamps

7. Classification of AR

8. Classification of AR Spatially Augmented Reality

9. Spatially Augmented Reality Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines ‘iLamps: Geometrically Aware and Self-Configurable Projectors’, SIGGRAPH 2003

14. Classification of AR Spatially Augmented Reality ShaderLamps

15. Shader Lamps Motivation View-dependent Appearance

18. Changing Appearance Projector Projector Virtual light source

28. Camera (and Projector) anatomy Camera center Image plane Principal point Principal axis

30. Changing Appearance Projector Projector Virtual light source

32. Radiance Adjustment Virtual L(x,  ) =  F(x,  ,  i ) L i (x,  i ) d  i I d Desired radiance BRDF Incident radiance

33. Radiance Adjustment k(x) cos(  p ) d(x) 2 Real L(x,  ) =  F(x,  ,  i ) L i (x,  i ) d  i L’(x,  ) = I d I p (x,  p ) Virtual Resultant radiance Pixel intensity

34. Radiance Adjustment d ( x ) 2 k ( x ) cos(  p ) I p (x,  p ) = I d , k ( x ) > 0 L ( x,  ) Virtual Real Intensity correction Desired radiance Pixel intensity Reflectance

36. Feathering in Overlap A B Projectors i d Traditional Solution A + B Projected Surface Weights A B

37. Feathering A B’ Projectors i d Traditional Solution A+B Projected Surface Weights B  A+B  A B

38. Feathering A B’ Projectors i d Traditional Solution A+B Projected Surface Weights A+B 

39. Occlusions A B Projectors h g d e f i A A B B B  A+B 

40. Occlusion Problems A B  Projectors h g d e f A+B  i Depth discontinuity A A B 

41. New Feathering A B Projectors h g d e f A+B  Overlap buffer 1 1 1 2 2 2 i Depth discontinuity Overlap 1 1 A A A A B B B  A+B  B 

42. New Feathering A B Projectors h g d e f A+B  A+B  Overlap buffer 1 1 1 2 2 2 i Depth discontinuity 1 1 A A A A B  B 

43. Virtual Illumination Shadows, Shading and Blending

48. Moving Objects Moving Surface Moving Projector Moving Viewer

49. Apparent Motion Ramesh Raskar, Remo Ziegler, Thomas Willwacher, “Cartoon Dioramas in Motion,” Proc. ACM Symposium on Nonphotorealistic Animation and Rendering (NPAR 2002)

50. Virtual Motion

52. ShaderLamps Virtual Reflectance Virtual Illumination Interaction Virtual Motion www.ShaderLamps.com

53. Projector-based Augmentation www.ShaderLamps.com Virtual Reflectance Virtual Illumination Interaction Virtual Motion

54. Desired Virtual Model © Andrei State Projected Guidance for Placement

56. Projector-based AR Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnação, L.M. ‘The Virtual Showcase’. IEEE Computer Graphics & Applications , vol. 21, no.6, 2001.

57. Steerable Projector Pinhanez et al 2003

59. Planar Multi-Projector Display [Raskar, Jeroen van Baar] 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 10 Seconds

63. Keystone Correction 1 . Compute screen to image homography 2 . Pre-warp input image Screen Projected image R Raskar

64. Automatic Keystone Correction with Camera and Tilt Sensor 1 . Camera and tilt sensor to find projector pose 2 . Compute screen to image homography 3 . Pre-warp input image Screen Projected image R Raskar [Raskar and Beardsley01]

65. Planar display surface Use homography ( A 3x3 ) User Single Projection Matrix ! V M i j A j = A i i = P T V ~ ~ P T j = [ A P T ] V ~

66. Ad-hoc Planar Cluster (Video) Self-contained Units, No centralized control No markers or cameras in environment Beyond the range of single camera

68. Quardic curved shape Displays Planetarium Sim/Viz Center Raskar, vanBaar, Willwacher, Rao ‘Quadric Transfer for Immersive Curved Displays’, EuroGraphics 2004

69. Curved projective transfer Quadric classification Projectively equivalent to sphere: hyperboloid of two sheets paraboloid sphere ellipsoid Ruled quadrics: hyperboloids of one sheet Degenerate ruled quadrics: cone two planes

70. Parametric Image Transfer X i j Planar Homography Quadric Transfer X i j

71. Overlap on Quadric Screens

75. Head Tracked Single Pass Rendering

77. Non-planar Display Video

78. What image should be projected ? Projector V : Virtual 3D Point M : Projected Point j User : T Screen ?

79. Step I : Calculate ‘desired’ image User V: Virtual 3D Point M i Desired Image Screen Projector

80. Step II : ‘Project’ the desired image from T Screen i Desired Image User M Projector

81. Step II : Render this scenario from P Screen V: Virtual 3D Point i j Projector Desired Image User Projected Image M

82. Result: Projecting a pre-warped image, so it looks correct Screen V: Virtual 3D Point i j Projector Desired Image User Projected Image M

83. AR with location-aware RFID

84. Ramesh Raskar, Paul Beardsley, Jeroen van Baar, Yao Wang, Paul Dietz, Johnny Lee, Darren Leigh, Thomas Willwacher Mitsubishi Electric Research Labs (MERL), Cambridge, MA R F I G Lamps : Interacting with a Self-describing World via Photosensing Wireless Tags and Projectors

85. Radio Frequency Identification Tags (RFID) microchip Antenna No batteries, Small size, Cost few cents

86. Warehousing Routing Library Baggage handling Currency Livestock tracking

87. Conventional Passive RFID

89. Prototype Tag RF tag + photosensor

90. Conventional RF tag Photo-sensing RF tag

92. Find tag location using handheld Projector Photosensing Wireless Tags Many geometric ops R F I R F I D Interactive stabilized projection (Radio Frequency Id & Geometry ) G Siggraph 2004

93. AR with Photosensing RFID and Handheld Projector

94. RFID (Radio Frequency Identification) RFI G (Radio Frequency Id and Geometry )

104. Change Detection without fixed camera, in any lighting condition Compare with new coordinates from a different view Record coordinates of tags from one view Before After

105. Finding Occlusions

106. Laser Guided Robot

107. Texture Adaptation

108. Desktop-like Interaction Selecting tags

109. Support for handheld projection

111. Image Quasi-Stabilization Eliminate hand jitter using inertial sensors+camera

112. Absolute Stabilization Image stays registered with world features

113. Image Stabilization

114. Interactive Projection

115. Adaptive Projection ‘ Copy and Paste’ Geometric and Photometric compensation

116. Prototype Handheld Projector

118. Robot ‘Laser’ Guidance Picking and Sorting Tagged Objects

122. Ramesh Raskar Mitsubishi Electric Research Labs (MERL), Cambridge, MA New Directions in Augmented Reality

123. Shader Lamps Motivation View-dependent Appearance

126. Virtual Motion

127. ShaderLamps Virtual Reflectance Virtual Illumination Interaction Virtual Motion www.ShaderLamps.com

128. Prototype Handheld Projector