Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Ramesh Raskar Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA New Directions in Augmented Reality
Parent Organization:  MELCO <ul><li>Mitsubishi Electric (MELCO), including subsidiaries </li></ul><ul><ul><li>$32B in annu...
MERL <ul><li>Mitsubishi Electric Research Laboratories </li></ul><ul><ul><li>Dick Waters, President & CEO </li></ul></ul><...
New Directions in Augmented Reality <ul><li>Spatially Augmented Reality:  </li></ul><ul><ul><li>Alternative approach </li>...
Non-trivial AR <ul><li>Painting with Light </li></ul><ul><ul><li>[Bandyopadhyay, Raskar, Fuchs 2001] </li></ul></ul>
Spatially Augmented Reality  (SAR) HMD-VR Spatially Immersive-VR AR using HMD Spatially Augmented Reality e.g. CAVE Video ...
Classification of AR
Classification of AR Spatially Augmented Reality
Spatially Augmented Reality Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines ‘iLamps: Geometrically Aware and Self-Co...
AR Issues <ul><li>Preprocessing: </li></ul><ul><ul><li>Authoring </li></ul></ul><ul><li>Runtime: </li></ul><ul><ul><li>Ide...
Advantages of Projectors <ul><ul><li>Size of image </li></ul></ul><ul><ul><li>Combination of images </li></ul></ul><ul><ul...
Disadvantages <ul><li>Projector limitations </li></ul><ul><ul><li>Limited depth of field </li></ul></ul><ul><ul><li>Shadow...
Advantages of Spatial Augmentation (SAR) <ul><ul><li>Augmentation of objects not view </li></ul></ul><ul><ul><li>Wide area...
Classification of AR Spatially Augmented Reality ShaderLamps
Shader Lamps Motivation View-dependent Appearance
Shader Lamps Image based Illumination <ul><ul><li>Basic Idea </li></ul></ul><ul><ul><ul><li>Render images and project on o...
 
Changing Appearance Projector Projector Virtual light source
Examples <ul><li>Son et Lumiere </li></ul><ul><ul><li>Projecting slide of augmented photo </li></ul></ul>
Examples <ul><li>Disney’s Haunted Mansion </li></ul><ul><ul><ul><li>Pre-recorded video </li></ul></ul></ul>Singing busts M...
<ul><li>Old </li></ul><ul><ul><li>Large,  rigid installations </li></ul></ul><ul><ul><li>A 2D image or  video projection <...
Challenges <ul><li>Complete illumination </li></ul><ul><ul><li>Image alignment </li></ul></ul><ul><ul><li>Special effects ...
Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul...
Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create virtual model,  G </li></ul></ul>...
Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model,  G </li></ul></ul></ul><ul...
Motivation <ul><li>Projector - a 3D projection device </li></ul><ul><ul><li>Projector is a  dual  of a camera </li></ul></...
Projector Model <ul><li>Pin hole model </li></ul><ul><ul><li>Equations for perspective projection </li></ul></ul><ul><ul><...
Camera (and Projector) anatomy Camera center Image plane Principal point Principal axis
Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul...
Changing Appearance Projector Projector Virtual light source
Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul...
Radiance Adjustment Virtual L(x,  ) =     F(x,   ,   i ) L i   (x,   i ) d  i I d Desired radiance BRDF Incident rad...
Radiance Adjustment k(x) cos(  p  )  d(x) 2 Real L(x,  ) =     F(x,   ,   i ) L i   (x,   i ) d  i L’(x,  ) = I d ...
Radiance Adjustment d ( x ) 2 k ( x )  cos(   p  ) I p  (x,   p ) = I d ,  k ( x ) > 0 L ( x,   ) Virtual Real Intensit...
<ul><ul><li>Rendering with </li></ul></ul><ul><ul><ul><li>Light at c.o.p. :  cos(  p ) </li></ul></ul></ul><ul><ul><ul><l...
Feathering in Overlap A B Projectors i d Traditional Solution A + B Projected Surface Weights A B
Feathering A B’ Projectors i d Traditional Solution A+B Projected Surface Weights B  A+B  A B
Feathering A B’ Projectors i d Traditional Solution A+B Projected Surface Weights A+B 
Occlusions A B Projectors h g d e f i A A B B B  A+B 
Occlusion Problems A B  Projectors h g d e f A+B  i Depth discontinuity A A B 
New Feathering A B Projectors h g d e f A+B  Overlap buffer 1 1 1 2 2 2 i Depth discontinuity Overlap 1 1 A A A A B B B ...
New Feathering A B Projectors h g d e f A+B  A+B  Overlap buffer 1 1 1 2 2 2 i Depth discontinuity 1 1 A A A A B  B 
Virtual Illumination Shadows, Shading and Blending
Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model,  G </li></ul></ul></ul><ul...
Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul...
Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul...
Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model,  G </li></ul></ul></ul><ul...
Moving Objects Moving Surface Moving Projector Moving Viewer
Apparent Motion Ramesh Raskar, Remo Ziegler, Thomas Willwacher, “Cartoon Dioramas in Motion,” Proc. ACM Symposium on Nonph...
Virtual Motion
Applications Indoors, under controlled lighting <ul><ul><li>Architectural models </li></ul></ul><ul><ul><ul><li>Augment wa...
ShaderLamps Virtual Reflectance Virtual Illumination Interaction Virtual Motion www.ShaderLamps.com
Projector-based Augmentation www.ShaderLamps.com Virtual Reflectance Virtual Illumination Interaction Virtual Motion
Desired Virtual Model © Andrei State Projected Guidance for Placement
 
Projector-based AR Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnação, L.M. ‘The Virtual Showcase’.  IEEE Computer ...
Steerable Projector Pinhanez et al 2003
Projection Techniques <ul><li>Projection Screen Geometries </li></ul><ul><ul><li>Planar </li></ul></ul><ul><ul><li>Rectili...
Planar Multi-Projector Display [Raskar, Jeroen van Baar] 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 10 Seconds
Planar projective transfer  What is homography ? <ul><ul><li>Two images of  3D points on a plane are related by a 3x3 matr...
Planar Homography (in 2D) <ul><ul><li>Two images of 3D points on a plane  </li></ul></ul><ul><ul><li>Related by a 3x3 matr...
Planar projective transfer  (homography) <ul><ul><li>Two images of  3D points on a plane are related by a 3x3 matrix </li>...
Keystone Correction 1 . Compute screen to image homography 2 . Pre-warp input image Screen Projected   image R Raskar
Automatic Keystone Correction with Camera and Tilt Sensor 1 . Camera and tilt sensor to find projector pose 2 . Compute sc...
Planar display surface Use homography ( A 3x3 ) User Single Projection Matrix ! V M i j A j  =  A   i i  =  P T  V ~ ~ P T...
Ad-hoc Planar Cluster (Video) Self-contained Units, No centralized control No markers or cameras in environment Beyond the...
Projection Techniques <ul><li>Projection Screen Geometries </li></ul><ul><ul><li>Planar </li></ul></ul><ul><ul><li>Rectili...
Quardic curved shape Displays Planetarium Sim/Viz Center Raskar, vanBaar, Willwacher, Rao ‘Quadric Transfer for Immersive ...
Curved projective transfer   Quadric classification Projectively equivalent to  sphere: hyperboloid of two sheets parabolo...
Parametric Image Transfer X i j Planar Homography Quadric Transfer X i j
Overlap on Quadric Screens
 
 
Vertex Shader for Quadric Transfer in Cg <ul><li>vertout main( appin IN,  uniform float4x4  modelViewProj ,  uniform float...
Head Tracked  Single Pass Rendering
Projection Techniques <ul><li>Projection Screen Geometries </li></ul><ul><ul><li>Planar </li></ul></ul><ul><ul><li>Rectili...
Non-planar Display Video
What image should be  projected ? Projector V : Virtual 3D Point M : Projected Point j User : T Screen ?
Step I :  Calculate ‘desired’ image User V: Virtual 3D Point M i Desired Image Screen Projector
Step II :  ‘Project’ the desired image from T Screen i Desired Image User M Projector
Step II :  Render this scenario from  P Screen V: Virtual 3D Point i j Projector Desired Image User Projected Image M
Result: Projecting a  pre-warped image, so it looks correct Screen V: Virtual 3D Point i j Projector Desired Image User Pr...
AR with location-aware RFID
Ramesh Raskar, Paul Beardsley, Jeroen van Baar, Yao Wang,  Paul Dietz, Johnny Lee, Darren Leigh, Thomas Willwacher   Mitsu...
Radio Frequency Identification Tags (RFID) microchip Antenna No batteries, Small size, Cost few cents
Warehousing Routing Library  Baggage  handling  Currency Livestock tracking
Conventional Passive RFID
Tagged Books in a Library <ul><li>Id  : List of books in RF range </li></ul><ul><li>No Precise Location Data   </li></ul><...
Prototype Tag RF tag  +  photosensor
Conventional RF tag Photo-sensing RF tag
 
Find tag location using   handheld Projector Photosensing  Wireless Tags Many geometric ops  R F I  R F I D Interactive st...
AR with Photosensing RFID and Handheld Projector
RFID (Radio  Frequency  Identification) RFI G (Radio  Frequency  Id  and  Geometry )
Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </...
Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </...
Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </...
Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </...
Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </...
Pattern MSB Pattern MSB-1 Pattern LSB <ul><li>For each tag </li></ul><ul><li>From light sequence, decode  x  and  y   coor...
Visual feedback of 2D position <ul><li>Receive via RF {( x 1 ,y 1 ), ( x 2 ,y 2 ), …} pixels </li></ul><ul><li>Illuminate ...
Visual feedback of 2D position <ul><li>Receive via RF {( x 1 ,y 1 ), ( x 2 ,y 2 ), …} pixels </li></ul><ul><li>Illuminate ...
3D from 2 Projector Views (Structure from Motion) <ul><li>Two+ unknown projector views </li></ul><ul><li>Correspondence is...
Change Detection  without fixed camera, in any lighting condition Compare with new coordinates from a different view   Rec...
Finding Occlusions
Laser Guided Robot
Texture Adaptation
Desktop-like Interaction Selecting tags
Support for handheld projection
Mouse Simulation <ul><li>Cursor follows handheld projector motion </li></ul><ul><li>Pre-warped image remains stable </li><...
Image Quasi-Stabilization Eliminate hand jitter using inertial sensors+camera
Absolute Stabilization Image stays registered with world features
Image Stabilization
Interactive Projection
Adaptive Projection ‘ Copy and Paste’ Geometric and Photometric compensation
Prototype Handheld Projector
Machine AR <ul><li>AR for cameras and machines </li></ul><ul><li>Face Dome  [Debevec 2001] </li></ul><ul><li>4D lighting  ...
Robot ‘Laser’ Guidance Picking and Sorting Tagged Objects
Acknowledgements <ul><li>MERL </li></ul><ul><ul><ul><li>Jeroen van Baar, Paul Beardsley, Remo Ziegler, Thomas Willwacher, ...
<ul><li>Projector-Camera Workshop </li></ul><ul><ul><li>June 2005 </li></ul></ul><ul><ul><li>www.PROCAMS.org </li></ul></u...
New Directions in Augmented Reality <ul><li>Spatially Augmented Reality:  </li></ul><ul><ul><li>Alternative approach </li>...
Ramesh Raskar Mitsubishi Electric Research Labs (MERL), Cambridge, MA New Directions in Augmented Reality
Shader Lamps Motivation View-dependent Appearance
Shader Lamps Image based Illumination <ul><ul><li>Basic Idea </li></ul></ul><ul><ul><ul><li>Render images and project on o...
 
Virtual Motion
ShaderLamps Virtual Reflectance Virtual Illumination Interaction Virtual Motion www.ShaderLamps.com
Prototype Handheld Projector
Upcoming SlideShare
Loading in …5
×

Svr Raskar

1,758 views

Published on

Published in: Technology, Art & Photos
  • Be the first to comment

  • Be the first to like this

Svr Raskar

  1. 1. Ramesh Raskar Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA New Directions in Augmented Reality
  2. 2. Parent Organization: MELCO <ul><li>Mitsubishi Electric (MELCO), including subsidiaries </li></ul><ul><ul><li>$32B in annual sales </li></ul></ul><ul><ul><li>125,000 people </li></ul></ul><ul><ul><li>10 business sectors: </li></ul></ul><ul><ul><ul><li>Space Development and Satellite Communications </li></ul></ul></ul><ul><ul><ul><li>Communications and Information Processing </li></ul></ul></ul><ul><ul><ul><li>Visual Equipment and Systems </li></ul></ul></ul><ul><ul><ul><li>Electronic Devices </li></ul></ul></ul><ul><ul><ul><li>Home Electronics </li></ul></ul></ul><ul><ul><ul><li>Energy </li></ul></ul></ul><ul><ul><ul><li>Industrial Equipment </li></ul></ul></ul><ul><ul><ul><li>Public-Use Systems </li></ul></ul></ul><ul><ul><ul><li>Transportation </li></ul></ul></ul>
  3. 3. MERL <ul><li>Mitsubishi Electric Research Laboratories </li></ul><ul><ul><li>Dick Waters, President & CEO </li></ul></ul><ul><li>Two labs, co-located in Cambridge, MA: </li></ul><ul><ul><li>MERL Technology Lab </li></ul></ul><ul><ul><ul><li>Kent Wittenburg, Director </li></ul></ul></ul><ul><ul><ul><li>Advanced systems development </li></ul></ul></ul><ul><ul><ul><li>~35 researchers </li></ul></ul></ul><ul><ul><li>MERL Research Lab </li></ul></ul><ul><ul><ul><li>Joe Marks, Director </li></ul></ul></ul><ul><ul><ul><li>Basic research in applied computing </li></ul></ul></ul><ul><ul><ul><li>23 researchers, interns, consultants </li></ul></ul></ul><ul><ul><ul><li>CV, HCI, Multimedia, CG, AI </li></ul></ul></ul>
  4. 4. New Directions in Augmented Reality <ul><li>Spatially Augmented Reality: </li></ul><ul><ul><li>Alternative approach </li></ul></ul><ul><li>Special effects in real world: </li></ul><ul><ul><li>Next big graphics challenge </li></ul></ul><ul><li>Location aware RFID: </li></ul><ul><ul><li>Converging technologies </li></ul></ul>www.raskar.com
  5. 5. Non-trivial AR <ul><li>Painting with Light </li></ul><ul><ul><li>[Bandyopadhyay, Raskar, Fuchs 2001] </li></ul></ul>
  6. 6. Spatially Augmented Reality (SAR) HMD-VR Spatially Immersive-VR AR using HMD Spatially Augmented Reality e.g. CAVE Video or Optical see-through SAR, Shaderlamps
  7. 7. Classification of AR
  8. 8. Classification of AR Spatially Augmented Reality
  9. 9. Spatially Augmented Reality Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines ‘iLamps: Geometrically Aware and Self-Configurable Projectors’, SIGGRAPH 2003
  10. 10. AR Issues <ul><li>Preprocessing: </li></ul><ul><ul><li>Authoring </li></ul></ul><ul><li>Runtime: </li></ul><ul><ul><li>Identification : Recognition of objects </li></ul></ul><ul><ul><ul><li>Using markers and visual tags </li></ul></ul></ul><ul><ul><li>Registration : Finding relative pose of display device </li></ul></ul><ul><ul><ul><li>Dynamic estimate of translation and rotation </li></ul></ul></ul><ul><ul><ul><li>Render/Warp images </li></ul></ul></ul><ul><ul><li>Interaction: </li></ul></ul><ul><ul><ul><li>Widgets, Gesture recognition, Visual feedback </li></ul></ul></ul>
  11. 11. Advantages of Projectors <ul><ul><li>Size of image </li></ul></ul><ul><ul><li>Combination of images </li></ul></ul><ul><ul><li>Shape of display surface </li></ul></ul>Image can be larger than device Images can be superimposed and added Displayed images may be non-planar
  12. 12. Disadvantages <ul><li>Projector limitations </li></ul><ul><ul><li>Limited depth of field </li></ul></ul><ul><ul><li>Shadows </li></ul></ul><ul><ul><li>Affected by display surface reflectance </li></ul></ul><ul><li>Challenges </li></ul><ul><ul><li>Calibration required </li></ul></ul><ul><ul><li>Rendering involves complex relationships </li></ul></ul>
  13. 13. Advantages of Spatial Augmentation (SAR) <ul><ul><li>Augmentation of objects not view </li></ul></ul><ul><ul><li>Wide area, High resolution </li></ul></ul><ul><ul><li>Comparison </li></ul></ul><ul><ul><li>Body-Worn Displays </li></ul></ul><ul><ul><ul><li>Better ergonomics </li></ul></ul></ul><ul><ul><ul><li>Reduced tracking requirements </li></ul></ul></ul><ul><ul><li>Hand-held Displays </li></ul></ul><ul><ul><ul><li>Avoids ‘last foot’ problem </li></ul></ul></ul>
  14. 14. Classification of AR Spatially Augmented Reality ShaderLamps
  15. 15. Shader Lamps Motivation View-dependent Appearance
  16. 16. Shader Lamps Image based Illumination <ul><ul><li>Basic Idea </li></ul></ul><ul><ul><ul><li>Render images and project on objects </li></ul></ul></ul><ul><ul><ul><li>Multiple projectors </li></ul></ul></ul><ul><ul><ul><li>View and object dependent color </li></ul></ul></ul>Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps: Animating Real Objects with Image Based Illumination,” Eurographics Rendering Worksop (EGRW 2001)
  17. 18. Changing Appearance Projector Projector Virtual light source
  18. 19. Examples <ul><li>Son et Lumiere </li></ul><ul><ul><li>Projecting slide of augmented photo </li></ul></ul>
  19. 20. Examples <ul><li>Disney’s Haunted Mansion </li></ul><ul><ul><ul><li>Pre-recorded video </li></ul></ul></ul>Singing busts Madame Leota
  20. 21. <ul><li>Old </li></ul><ul><ul><li>Large, rigid installations </li></ul></ul><ul><ul><li>A 2D image or video projection </li></ul></ul><ul><ul><li>Single projector </li></ul></ul><ul><ul><li>Texture </li></ul></ul><ul><li>New </li></ul><ul><ul><li>Easy setup, Non-trivial objects </li></ul></ul><ul><ul><li>Real time 3D animation </li></ul></ul><ul><ul><li>Multiple projectors </li></ul></ul><ul><ul><li>BRDF </li></ul></ul>
  21. 22. Challenges <ul><li>Complete illumination </li></ul><ul><ul><li>Image alignment </li></ul></ul><ul><ul><li>Special effects </li></ul></ul><ul><ul><ul><li>Changing appearance and lighting </li></ul></ul></ul><ul><ul><li>Complex geometry, self-occlusions </li></ul></ul><ul><ul><li>Merging multiple projectors </li></ul></ul>
  22. 23. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul><ul><li>Approximately position projector(s) </li></ul></ul></ul><ul><ul><ul><li>Compute pose, P </li></ul></ul></ul><ul><ul><ul><ul><li>Find features </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Find pixels that illuminate them </li></ul></ul></ul></ul><ul><ul><ul><li>Compute intensity correction </li></ul></ul></ul><ul><ul><li>Run time </li></ul></ul><ul><ul><ul><li>Render images of 3D model </li></ul></ul></ul><ul><ul><ul><li>Intensity correction for object shape </li></ul></ul></ul><ul><ul><ul><li>Feathering for projector overlap </li></ul></ul></ul>
  23. 24. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create virtual model, G </li></ul></ul></ul>Faro arm
  24. 25. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model, G </li></ul></ul></ul><ul><ul><ul><li>Approximately position projector(s) </li></ul></ul></ul><ul><ul><ul><li>Find pose, P </li></ul></ul></ul>
  25. 26. Motivation <ul><li>Projector - a 3D projection device </li></ul><ul><ul><li>Projector is a dual of a camera </li></ul></ul><ul><ul><li>Relates 3D space and image in framebuffer </li></ul></ul><ul><ul><li>A useful abstraction : geometric projection model </li></ul></ul>
  26. 27. Projector Model <ul><li>Pin hole model </li></ul><ul><ul><li>Equations for perspective projection </li></ul></ul><ul><ul><li>Relationship between 3D and 2D </li></ul></ul><ul><ul><li>Intrinsic and Extrinsic Parameters </li></ul></ul>
  27. 28. Camera (and Projector) anatomy Camera center Image plane Principal point Principal axis
  28. 29. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul><ul><li>Approximately position projector(s) </li></ul></ul></ul><ul><ul><ul><li>Compute pose, P </li></ul></ul></ul><ul><ul><ul><ul><li>Find fiducials </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Find pixels that illuminate them </li></ul></ul></ul></ul><ul><ul><ul><li>Compute intensity correction </li></ul></ul></ul><ul><ul><li>Run time </li></ul></ul><ul><ul><ul><li>Render images of 3D model using matrix P </li></ul></ul></ul><ul><ul><ul><li>Intensity correction for object shape </li></ul></ul></ul><ul><ul><ul><li>Feathering for projector overlap </li></ul></ul></ul>
  29. 30. Changing Appearance Projector Projector Virtual light source
  30. 31. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul><ul><li>Approximately position projector(s) </li></ul></ul></ul><ul><ul><ul><li>Compute pose, P </li></ul></ul></ul><ul><ul><ul><ul><li>Find fiducials </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Find pixels that illuminate them </li></ul></ul></ul></ul><ul><ul><ul><li>Compute intensity correction </li></ul></ul></ul><ul><ul><li>Run time </li></ul></ul><ul><ul><ul><li>Render images of 3D model using matrix P </li></ul></ul></ul><ul><ul><ul><li>Intensity correction for object shape </li></ul></ul></ul><ul><ul><ul><li>Feathering for projector overlap </li></ul></ul></ul>
  31. 32. Radiance Adjustment Virtual L(x,  ) =  F(x,  ,  i ) L i (x,  i ) d  i I d Desired radiance BRDF Incident radiance
  32. 33. Radiance Adjustment k(x) cos(  p ) d(x) 2 Real L(x,  ) =  F(x,  ,  i ) L i (x,  i ) d  i L’(x,  ) = I d I p (x,  p ) Virtual Resultant radiance Pixel intensity
  33. 34. Radiance Adjustment d ( x ) 2 k ( x ) cos(  p ) I p (x,  p ) = I d , k ( x ) > 0 L ( x,  ) Virtual Real Intensity correction Desired radiance Pixel intensity Reflectance
  34. 35. <ul><ul><li>Rendering with </li></ul></ul><ul><ul><ul><li>Light at c.o.p. : cos(  p ) </li></ul></ul></ul><ul><ul><ul><li>Diffuse reflectance : k </li></ul></ul></ul><ul><ul><ul><li>Distance attenuation : 1/ d (x) 2 </li></ul></ul></ul><ul><ul><ul><li> p > 60 0 cut off </li></ul></ul></ul>Intensity Correction Per-pixel factor x L'   P I P d Rendered Image d ( x ) 2 k ( x ) cos(  p ) I p (x,  p ) = L ( x,  )
  35. 36. Feathering in Overlap A B Projectors i d Traditional Solution A + B Projected Surface Weights A B
  36. 37. Feathering A B’ Projectors i d Traditional Solution A+B Projected Surface Weights B  A+B  A B
  37. 38. Feathering A B’ Projectors i d Traditional Solution A+B Projected Surface Weights A+B 
  38. 39. Occlusions A B Projectors h g d e f i A A B B B  A+B 
  39. 40. Occlusion Problems A B  Projectors h g d e f A+B  i Depth discontinuity A A B 
  40. 41. New Feathering A B Projectors h g d e f A+B  Overlap buffer 1 1 1 2 2 2 i Depth discontinuity Overlap 1 1 A A A A B B B  A+B  B 
  41. 42. New Feathering A B Projectors h g d e f A+B  A+B  Overlap buffer 1 1 1 2 2 2 i Depth discontinuity 1 1 A A A A B  B 
  42. 43. Virtual Illumination Shadows, Shading and Blending
  43. 44. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model, G </li></ul></ul></ul><ul><ul><ul><li>Approximately position projector(s) </li></ul></ul></ul><ul><ul><ul><li>Compute pose, P </li></ul></ul></ul><ul><ul><ul><li>Compute intensity correction,  </li></ul></ul></ul><ul><ul><li>Run time </li></ul></ul><ul><ul><ul><li>Render image [ I ] using model G from pose P </li></ul></ul></ul>
  44. 45. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul><ul><li>Approximately position projector(s) </li></ul></ul></ul><ul><ul><ul><li>Compute pose </li></ul></ul></ul><ul><ul><ul><li>Compute intensity correction </li></ul></ul></ul><ul><ul><li>Run time </li></ul></ul><ul><ul><ul><li>Render image [ I ] </li></ul></ul></ul><ul><ul><ul><li>Apply intensity correction for object shape [  ] * [ I ] </li></ul></ul></ul><ul><ul><ul><li>Apply feathering for projector overlap, [  ] * [  ] * [ I ] </li></ul></ul></ul>
  45. 46. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul><ul><li>Approximately position projector(s) </li></ul></ul></ul><ul><ul><ul><li>Compute pose </li></ul></ul></ul><ul><ul><ul><li>Compute intensity correction </li></ul></ul></ul><ul><ul><li>Run time </li></ul></ul><ul><ul><ul><li>Render image [ I ] </li></ul></ul></ul><ul><ul><ul><li>Intensity correction, [  ] * [ I ] </li></ul></ul></ul><ul><ul><ul><li>Feathering, [  ] * [  ] * [ I ] </li></ul></ul></ul>
  46. 47. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model, G </li></ul></ul></ul><ul><ul><ul><li>Approximately position projector(s) </li></ul></ul></ul><ul><ul><ul><li>Compute pose, P </li></ul></ul></ul><ul><ul><ul><ul><li>Find fiducials </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Find pixels that illuminate them </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Find projector pose </li></ul></ul></ul></ul><ul><ul><li>Run time </li></ul></ul><ul><ul><ul><li>Render 3D model G from P , [ I ] </li></ul></ul></ul><ul><ul><ul><li>Intensity correction for object shape [  ] * [ I ] </li></ul></ul></ul><ul><ul><ul><li>Feathering for projector overlap [  ] * [  ] * [ I ] </li></ul></ul></ul>
  47. 48. Moving Objects Moving Surface Moving Projector Moving Viewer
  48. 49. Apparent Motion Ramesh Raskar, Remo Ziegler, Thomas Willwacher, “Cartoon Dioramas in Motion,” Proc. ACM Symposium on Nonphotorealistic Animation and Rendering (NPAR 2002)
  49. 50. Virtual Motion
  50. 51. Applications Indoors, under controlled lighting <ul><ul><li>Architectural models </li></ul></ul><ul><ul><ul><li>Augment walk-around scaled model of buildings </li></ul></ul></ul><ul><ul><ul><li>Project and ‘paint’ surface colors, textures </li></ul></ul></ul><ul><ul><ul><li>Lighting, sunlight, seasons </li></ul></ul></ul><ul><ul><ul><li>Internal structure, pipes, wiring </li></ul></ul></ul><ul><ul><li>Assembly line </li></ul></ul><ul><ul><ul><li>Instructional text, images and procedures </li></ul></ul></ul><ul><ul><li>Entertainment </li></ul></ul><ul><ul><ul><li>Live shows, exhibits, demonstrations </li></ul></ul></ul>
  51. 52. ShaderLamps Virtual Reflectance Virtual Illumination Interaction Virtual Motion www.ShaderLamps.com
  52. 53. Projector-based Augmentation www.ShaderLamps.com Virtual Reflectance Virtual Illumination Interaction Virtual Motion
  53. 54. Desired Virtual Model © Andrei State Projected Guidance for Placement
  54. 56. Projector-based AR Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnação, L.M. ‘The Virtual Showcase’. IEEE Computer Graphics & Applications , vol. 21, no.6, 2001.
  55. 57. Steerable Projector Pinhanez et al 2003
  56. 58. Projection Techniques <ul><li>Projection Screen Geometries </li></ul><ul><ul><li>Planar </li></ul></ul><ul><ul><li>Rectilinear </li></ul></ul><ul><ul><li>Cylindrical </li></ul></ul><ul><ul><li>Spherical </li></ul></ul><ul><ul><li>Irregular </li></ul></ul>Planar Homography Quadric image transfer Discretized Warping
  57. 59. Planar Multi-Projector Display [Raskar, Jeroen van Baar] 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 10 Seconds
  58. 60. Planar projective transfer What is homography ? <ul><ul><li>Two images of 3D points on a plane are related by a 3x3 matrix </li></ul></ul>Screen Camera 1 M i j Camera 2
  59. 61. Planar Homography (in 2D) <ul><ul><li>Two images of 3D points on a plane </li></ul></ul><ul><ul><li>Related by a 3x3 matrix </li></ul></ul>a1 a2 a3 b1 b2 b3 c1 c2 c3 Proj 1 A 3 x 3 Proj 2 i j j x j y 1 j = A 3 x 3 i ~ = i x i y 1 k j x = ( a • i ) / ( c • i ) j y = ( b • i ) / ( c • i )
  60. 62. Planar projective transfer (homography) <ul><ul><li>Two images of 3D points on a plane are related by a 3x3 matrix </li></ul></ul>Screen Camera 1 M i j Camera 2 Defined by 4 or more corresponding pixels
  61. 63. Keystone Correction 1 . Compute screen to image homography 2 . Pre-warp input image Screen Projected image R Raskar
  62. 64. Automatic Keystone Correction with Camera and Tilt Sensor 1 . Camera and tilt sensor to find projector pose 2 . Compute screen to image homography 3 . Pre-warp input image Screen Projected image R Raskar [Raskar and Beardsley01]
  63. 65. Planar display surface Use homography ( A 3x3 ) User Single Projection Matrix ! V M i j A j = A i i = P T V ~ ~ P T j = [ A P T ] V ~
  64. 66. Ad-hoc Planar Cluster (Video) Self-contained Units, No centralized control No markers or cameras in environment Beyond the range of single camera
  65. 67. Projection Techniques <ul><li>Projection Screen Geometries </li></ul><ul><ul><li>Planar </li></ul></ul><ul><ul><li>Rectilinear </li></ul></ul><ul><ul><li>Cylindrical </li></ul></ul><ul><ul><li>Spherical </li></ul></ul><ul><ul><li>Irregular </li></ul></ul>Planar Homography Quadric image transfer Discretized Warping
  66. 68. Quardic curved shape Displays Planetarium Sim/Viz Center Raskar, vanBaar, Willwacher, Rao ‘Quadric Transfer for Immersive Curved Displays’, EuroGraphics 2004
  67. 69. Curved projective transfer Quadric classification Projectively equivalent to sphere: hyperboloid of two sheets paraboloid sphere ellipsoid Ruled quadrics: hyperboloids of one sheet Degenerate ruled quadrics: cone two planes
  68. 70. Parametric Image Transfer X i j Planar Homography Quadric Transfer X i j
  69. 71. Overlap on Quadric Screens
  70. 74. Vertex Shader for Quadric Transfer in Cg <ul><li>vertout main( appin IN, uniform float4x4 modelViewProj , uniform float4 constColor, uniform float3x3 A , uniform float3x3 E , uniform float3 e ) { </li></ul><ul><li>vertout OUT; </li></ul><ul><li>float4 m1 = float4(IN.position.x, IN.position.y, IN.position.z, 1.0f ); </li></ul><ul><li>float4 m, mi ; float3 m2,mp; float scale; </li></ul><ul><li>  </li></ul><ul><li>m = mul( modelViewProj , m1); </li></ul><ul><li>m2.x = m.x/m.w; m2.y = m.y/m.w; m2.z = 1; </li></ul><ul><li>scale = mul(m2, mul( E ,m2)); </li></ul><ul><li>mp = mul( A ,m2) + sqrt(scale)* e ; </li></ul><ul><li>mi.x = m.w * (mp.x)/(mp.z); </li></ul><ul><li>mi.y = m.w * (mp.y)/(mp.z); </li></ul><ul><li>mi.zw = m.zw; </li></ul><ul><li>OUT.position = mi;  </li></ul><ul><li>OUT.color0 = IN.color0; // Use the original per-vertex color specified </li></ul><ul><li>return OUT; </li></ul><ul><li>} </li></ul><ul><li>(Code in Course Notes) </li></ul>
  71. 75. Head Tracked Single Pass Rendering
  72. 76. Projection Techniques <ul><li>Projection Screen Geometries </li></ul><ul><ul><li>Planar </li></ul></ul><ul><ul><li>Rectilinear </li></ul></ul><ul><ul><li>Cylindrical </li></ul></ul><ul><ul><li>Spherical </li></ul></ul><ul><ul><li>Irregular </li></ul></ul>Planar Homography Quadric image transfer Discretized Warping
  73. 77. Non-planar Display Video
  74. 78. What image should be projected ? Projector V : Virtual 3D Point M : Projected Point j User : T Screen ?
  75. 79. Step I : Calculate ‘desired’ image User V: Virtual 3D Point M i Desired Image Screen Projector
  76. 80. Step II : ‘Project’ the desired image from T Screen i Desired Image User M Projector
  77. 81. Step II : Render this scenario from P Screen V: Virtual 3D Point i j Projector Desired Image User Projected Image M
  78. 82. Result: Projecting a pre-warped image, so it looks correct Screen V: Virtual 3D Point i j Projector Desired Image User Projected Image M
  79. 83. AR with location-aware RFID
  80. 84. Ramesh Raskar, Paul Beardsley, Jeroen van Baar, Yao Wang, Paul Dietz, Johnny Lee, Darren Leigh, Thomas Willwacher Mitsubishi Electric Research Labs (MERL), Cambridge, MA R F I G Lamps : Interacting with a Self-describing World via Photosensing Wireless Tags and Projectors
  81. 85. Radio Frequency Identification Tags (RFID) microchip Antenna No batteries, Small size, Cost few cents
  82. 86. Warehousing Routing Library Baggage handling Currency Livestock tracking
  83. 87. Conventional Passive RFID
  84. 88. Tagged Books in a Library <ul><li>Id : List of books in RF range </li></ul><ul><li>No Precise Location Data </li></ul><ul><li>Are books in sorted order ? </li></ul><ul><li>Which book is upside down ? </li></ul>
  85. 89. Prototype Tag RF tag + photosensor
  86. 90. Conventional RF tag Photo-sensing RF tag
  87. 92. Find tag location using handheld Projector Photosensing Wireless Tags Many geometric ops R F I R F I D Interactive stabilized projection (Radio Frequency Id & Geometry ) G Siggraph 2004
  88. 93. AR with Photosensing RFID and Handheld Projector
  89. 94. RFID (Radio Frequency Identification) RFI G (Radio Frequency Id and Geometry )
  90. 95. Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </li></ul><ul><li>Tags decode temporal code </li></ul>
  91. 96. Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </li></ul><ul><li>Tags decode temporal code </li></ul>
  92. 97. Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </li></ul><ul><li>Tags decode temporal code </li></ul>
  93. 98. Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </li></ul><ul><li>Tags decode temporal code </li></ul>
  94. 99. Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </li></ul><ul><li>Tags decode temporal code </li></ul>
  95. 100. Pattern MSB Pattern MSB-1 Pattern LSB <ul><li>For each tag </li></ul><ul><li>From light sequence, decode x and y coordinate </li></ul><ul><li>Transmit back to RF reader ( Id , x, y ) </li></ul>0 1 1 0 0 X=12
  96. 101. Visual feedback of 2D position <ul><li>Receive via RF {( x 1 ,y 1 ), ( x 2 ,y 2 ), …} pixels </li></ul><ul><li>Illuminate those positions </li></ul>
  97. 102. Visual feedback of 2D position <ul><li>Receive via RF {( x 1 ,y 1 ), ( x 2 ,y 2 ), …} pixels </li></ul><ul><li>Illuminate those positions </li></ul>
  98. 103. 3D from 2 Projector Views (Structure from Motion) <ul><li>Two+ unknown projector views </li></ul><ul><li>Correspondence is trivial </li></ul><ul><li>Applications </li></ul><ul><ul><li>Detect 3D deformations </li></ul></ul><ul><ul><li>Trajectory grouping </li></ul></ul>
  99. 104. Change Detection without fixed camera, in any lighting condition Compare with new coordinates from a different view Record coordinates of tags from one view Before After
  100. 105. Finding Occlusions
  101. 106. Laser Guided Robot
  102. 107. Texture Adaptation
  103. 108. Desktop-like Interaction Selecting tags
  104. 109. Support for handheld projection
  105. 110. Mouse Simulation <ul><li>Cursor follows handheld projector motion </li></ul><ul><li>Pre-warped image remains stable </li></ul>
  106. 111. Image Quasi-Stabilization Eliminate hand jitter using inertial sensors+camera
  107. 112. Absolute Stabilization Image stays registered with world features
  108. 113. Image Stabilization
  109. 114. Interactive Projection
  110. 115. Adaptive Projection ‘ Copy and Paste’ Geometric and Photometric compensation
  111. 116. Prototype Handheld Projector
  112. 117. Machine AR <ul><li>AR for cameras and machines </li></ul><ul><li>Face Dome [Debevec 2001] </li></ul><ul><li>4D lighting [MPI, MERL] </li></ul>
  113. 118. Robot ‘Laser’ Guidance Picking and Sorting Tagged Objects
  114. 119. Acknowledgements <ul><li>MERL </li></ul><ul><ul><ul><li>Jeroen van Baar, Paul Beardsley, Remo Ziegler, Thomas Willwacher, Srinivas Rao, Cliff Forlines, Paul Dietz, Johnny Lee </li></ul></ul></ul><ul><li>Office of the Future group at UNC Chapel Hill </li></ul><ul><ul><ul><li>Greg Welch, Kok-lim Low, Deepak B’padhyay </li></ul></ul></ul><ul><ul><ul><li>Henry Fuchs, Herman Towles </li></ul></ul></ul><ul><ul><ul><li>Wei-chao Chen </li></ul></ul></ul><ul><li>Mitsubishi Electric, Japan </li></ul><ul><ul><ul><li>Yoshihiro Ashizaki, Masatoshi Kameyama, Masato Ogata, Keiichi Shiotani </li></ul></ul></ul><ul><li>Images </li></ul><ul><ul><ul><li>Oliver Bimber, Claudio Pinhanez </li></ul></ul></ul><ul><ul><ul><li>Ed Lantz (Spitz Inc.), Marc Pollefeys (UNC Chapel Hill) </li></ul></ul></ul><ul><ul><ul><li>Aditi Majumder, Rahul Sukthankar,, Chris Jaynes, Shree Nayar </li></ul></ul></ul><ul><ul><ul><li>Apologies </li></ul></ul></ul><ul><ul><ul><ul><li>(Not able to include work by others) </li></ul></ul></ul></ul><ul><li>SVR, Absolut, Luciano Peres </li></ul>
  115. 120. <ul><li>Projector-Camera Workshop </li></ul><ul><ul><li>June 2005 </li></ul></ul><ul><ul><li>www.PROCAMS.org </li></ul></ul><ul><li>Web Page : </li></ul><ul><li>http://raskar.com/Projector/ </li></ul><ul><li>Projector mailing list </li></ul><ul><ul><ul><li>[email_address] </li></ul></ul></ul><ul><ul><ul><li>“ subscribe projector” </li></ul></ul></ul>
  116. 121. New Directions in Augmented Reality <ul><li>Spatially Augmented Reality: </li></ul><ul><ul><li>Alternative approach </li></ul></ul><ul><li>Special effects in real world: </li></ul><ul><ul><li>Next big graphics challenge </li></ul></ul><ul><li>Location aware RFID: </li></ul><ul><ul><li>Converging technologies </li></ul></ul>www.raskar.com
  117. 122. Ramesh Raskar Mitsubishi Electric Research Labs (MERL), Cambridge, MA New Directions in Augmented Reality
  118. 123. Shader Lamps Motivation View-dependent Appearance
  119. 124. Shader Lamps Image based Illumination <ul><ul><li>Basic Idea </li></ul></ul><ul><ul><ul><li>Render images and project on objects </li></ul></ul></ul><ul><ul><ul><li>Multiple projectors </li></ul></ul></ul><ul><ul><ul><li>View and object dependent color </li></ul></ul></ul>Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps: Animating Real Objects with Image Based Illumination,” Eurographics Rendering Worksop (EGRW 2001)
  120. 126. Virtual Motion
  121. 127. ShaderLamps Virtual Reflectance Virtual Illumination Interaction Virtual Motion www.ShaderLamps.com
  122. 128. Prototype Handheld Projector

×