Raskar Graphics Interface May05 Web

645 views

Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Raskar Graphics Interface May05 Web

  1. 1. Ramesh Raskar Mitsubishi Electric Research Labs (MERL) Cambridge, MA Special Effects in the Real World
  2. 2. Special Effects <ul><ul><li>Photorealism </li></ul></ul><ul><ul><ul><li>Stays on screens </li></ul></ul></ul><ul><ul><ul><li>Yet to affect daily life in real time </li></ul></ul></ul><ul><ul><ul><ul><li>Entertainment and Visualization </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Unlike other fields of CS </li></ul></ul></ul></ul>
  3. 3. Changing Appearance Virtual light source
  4. 4. Changing Virtual Illumination
  5. 6. Special Effects in the Real World <ul><ul><li>Photorealism </li></ul></ul><ul><ul><ul><li>Limited to display screens </li></ul></ul></ul><ul><ul><ul><li>Yet to affect daily life in real time </li></ul></ul></ul><ul><ul><ul><ul><li>Entertainment and Visualization </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Unlike other fields of CS </li></ul></ul></ul></ul><ul><ul><li>Fusion: real world with graphics </li></ul></ul><ul><ul><ul><li>Next big challenge in CG ? </li></ul></ul></ul><ul><ul><ul><li>Believable co-existence </li></ul></ul></ul>
  6. 7. Changing Appearance Projector Projector Virtual light source
  7. 8. Changing Virtual Illumination Raskar, Welch, Low, Bandyopadhyay, “ Shader Lamps ” (2000)
  8. 9. Virtual Motion
  9. 10. Virtual Motion Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)
  10. 11. Dynamic Augmentation Projecting on Tracked Objects
  11. 12. ShaderLamps Virtual Reflectance Virtual Illumination Interaction Virtual Motion www.ShaderLamps.com
  12. 13. Projector-based Augmentation www.ShaderLamps.com Virtual Reflectance Virtual Illumination Interaction Virtual Motion
  13. 14. Poor Man’s Palace
  14. 15. Maya: World is an Illusion <ul><li>Fake World </li></ul><ul><ul><li>We all live in one </li></ul></ul><ul><li>Real-life Fakes </li></ul><ul><ul><li>Not just photos and videos but surroundings </li></ul></ul><ul><li>Privacy </li></ul>
  15. 16. Augmented Reality (AR) With See-through Head-mounted Display © Andrei State
  16. 17. Head-attached Body-attached spatial Spatial Augmented Reality Classification of Augmented Reality hand-held display spatial see-through display head- mounted display retinal display projector projector projector
  17. 18. Being There <ul><li>Recreate Large Environments </li></ul><ul><ul><li>‘ BeingThere’, walk-around </li></ul></ul><ul><ul><li>Museums, Exhibitions, Training </li></ul></ul>Kok-Lim Low, Greg Welch, Anselmo Lastra, Henry Fuchs. “Life-Sized Projector-Based Dioramas,” Proc. ACM Symposium on Virtual Reality Software and Technology 2001 (VRST 2001), November 2001.
  18. 19. Desired Virtual Model © Andrei State Projected Guidance for Placement
  19. 21. Augmented Reality Head-mounted Display Projectors © Andrei State
  20. 22. Opportunities <ul><ul><li>Information Augmentation </li></ul></ul><ul><ul><ul><li>Office, factory or assembly Training </li></ul></ul></ul><ul><ul><li>Increasing value of real products </li></ul></ul><ul><ul><ul><li>Smart illumination to enhance appearance of object </li></ul></ul></ul><ul><ul><li>Entertainment </li></ul></ul><ul><ul><ul><li>Animating objects, Museums, Special Effects, Laser Shows </li></ul></ul></ul><ul><ul><li>Advertising </li></ul></ul><ul><ul><ul><li>Car showrooms, Furniture store </li></ul></ul></ul><ul><ul><li>Design and Ergonomics </li></ul></ul><ul><ul><ul><li>Protyping products, visualize material changes </li></ul></ul></ul>Projector + Camera
  21. 23. <ul><li>Why </li></ul><ul><ul><li>Special Effects in the Real World </li></ul></ul><ul><ul><li>Photorealism for everyday </li></ul></ul><ul><li>Tools for Augmentation </li></ul><ul><ul><li>Projectors, Sensors, RFID </li></ul></ul><ul><li>Illusion </li></ul><ul><ul><li>Beyond Visual Senses </li></ul></ul><ul><ul><li>Beyond Projectors </li></ul></ul><ul><ul><li>Beyond Gouraud </li></ul></ul>Spatially Augmented Reality
  22. 24. Spatially Augmented Reality (SAR) HMD-VR Spatially Immersive-VR AR using HMD Spatially Augmented Reality e.g. CAVE Video or Optical see-through SAR, Shaderlamps
  23. 25. AR Issues <ul><li>Preprocessing: </li></ul><ul><ul><li>Authoring </li></ul></ul><ul><li>Runtime: </li></ul><ul><ul><li>Identification : Recognition of objects </li></ul></ul><ul><ul><ul><li>Using markers and visual tags </li></ul></ul></ul><ul><ul><li>Registration : Finding relative pose of display device </li></ul></ul><ul><ul><ul><li>Dynamic estimate of translation and rotation </li></ul></ul></ul><ul><ul><ul><li>Render/Warp images </li></ul></ul></ul><ul><ul><li>Interaction: </li></ul></ul><ul><ul><ul><li>Widgets, Gesture recognition, Visual feedback </li></ul></ul></ul>
  24. 26. Spatially Augmented Reality Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines ‘iLamps: Geometrically Aware and Self-Configurable Projectors’, SIGGRAPH 2003
  25. 27. AR Issues <ul><li>Preprocessing: </li></ul><ul><ul><li>Authoring </li></ul></ul><ul><li>Runtime: </li></ul><ul><ul><li>Identification : Recognition of objects </li></ul></ul><ul><ul><ul><li>Using markers and visual tags </li></ul></ul></ul><ul><ul><li>Registration : Finding relative pose of display device </li></ul></ul><ul><ul><ul><li>Dynamic estimate of translation and rotation </li></ul></ul></ul><ul><ul><ul><li>Render/Warp images </li></ul></ul></ul><ul><ul><li>Interaction: </li></ul></ul><ul><ul><ul><li>Widgets, Gesture recognition, Visual feedback </li></ul></ul></ul><ul><ul><li>Photorealism ? </li></ul></ul>RFID Cameras Trackers
  26. 28. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object </li></ul></ul></ul><ul><ul><ul><li>Roughly align projectors </li></ul></ul></ul><ul><ul><ul><li>Compute correction </li></ul></ul></ul><ul><ul><li>Run time </li></ul></ul><ul><ul><ul><li>Render images of 3D model </li></ul></ul></ul><ul><ul><ul><li>Correct </li></ul></ul></ul>
  27. 29. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create virtual model </li></ul></ul></ul>Faro arm
  28. 30. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul><ul><li>Approximately position projector(s) </li></ul></ul></ul><ul><ul><ul><li>Calibrate by finding pose </li></ul></ul></ul>
  29. 31. Steps <ul><ul><li>Preprocessing </li></ul></ul><ul><ul><ul><li>Scan 3D object and create model </li></ul></ul></ul><ul><ul><ul><li>Approximately position projector(s) </li></ul></ul></ul><ul><ul><ul><li>Calibrate by finding pose </li></ul></ul></ul><ul><ul><li>Run-time </li></ul></ul><ul><ul><ul><li>Inverse Rendering of 3D Models </li></ul></ul></ul>
  30. 32. Warehousing Routing Library Baggage handling Currency Livestock tracking RFID
  31. 33. Conventional Passive RFID
  32. 34. Tagged Books in a Library <ul><li>Id : List of books in RF range </li></ul><ul><li>No Precise Location Data </li></ul><ul><li>Are books in sorted order ? </li></ul><ul><li>Which book is upside down ? </li></ul>
  33. 35. Handheld Projector iLamps 2002 RFIG Lamps 2003-04 Pocket Projector 2004-05
  34. 36. Mitsubishi Pocket Projector
  35. 38. Prototype Tag RF tag + photosensor
  36. 39. Conventional RF tag Photo-sensing RF tag
  37. 41. Find tag location using handheld Projector Photosensing Wireless Tags Many geometric ops R F I R F I D Interactive stabilized projection (Radio Frequency Id & Geometry) G Siggraph 2004
  38. 42. AR with Photosensing RFID and Handheld Projector
  39. 43. Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </li></ul><ul><li>Tags decode temporal code </li></ul>
  40. 44. Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </li></ul><ul><li>Tags decode temporal code </li></ul>
  41. 45. Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </li></ul><ul><li>Tags decode temporal code </li></ul>
  42. 46. Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </li></ul><ul><li>Tags decode temporal code </li></ul>
  43. 47. Pattern MSB Pattern MSB-1 Pattern LSB Projected Sequential Frames <ul><li>Handheld Projector beams binary coded stripes </li></ul><ul><li>Tags decode temporal code </li></ul>
  44. 48. Pattern MSB Pattern MSB-1 Pattern LSB <ul><li>For each tag </li></ul><ul><li>From light sequence, decode x and y coordinate </li></ul><ul><li>Transmit back to RF reader (Id, x, y) </li></ul>0 1 1 0 0 X=12
  45. 49. Visual feedback of 2D position <ul><li>Receive via RF {( x 1 ,y 1 ), ( x 2 ,y 2 ), …} pixels </li></ul><ul><li>Illuminate those positions </li></ul>
  46. 50. RFID (Radio Frequency Identification) RFI G (Radio Frequency Id and Geometry )
  47. 51. Projectors and Surfaces Curved Planar Non-planar Single Projector Multiple Projectors Pocket-Proj Objects 2000 2000 2002 2001 1999 2002 2003 1999 1998 Projector j User : T ?
  48. 52. Pieces of the Puzzle Actuated Surfaces Actuated Workbench [Pangaro, Maynes-Aminzade, Ishii UIST 2002]
  49. 53. <ul><li>Why </li></ul><ul><ul><li>Special Effects in the Real World </li></ul></ul><ul><ul><li>Photorealism for everyday </li></ul></ul><ul><li>Tools for Augmentation </li></ul><ul><ul><li>Projectors, Sensors, RFID </li></ul></ul><ul><li>Illusion </li></ul><ul><ul><li>Beyond Visual Senses </li></ul></ul><ul><ul><li>Beyond Projectors </li></ul></ul><ul><ul><li>Beyond Gouraud </li></ul></ul>Spatially Augmented Reality
  50. 54. Believable Co-existence <ul><li>Other Senses </li></ul><ul><ul><li>Haptic, Smell, Taste, Locomotion </li></ul></ul><ul><li>Beyond Gouraud Effects </li></ul><ul><ul><li>Complex BRDF </li></ul></ul><ul><li>Display Choices </li></ul><ul><ul><li>Polymers, e-Ink </li></ul></ul>
  51. 55. Augmentation of “five senses” <ul><li>Visual Displays </li></ul><ul><li>Auditory Displays </li></ul><ul><li>Haptic Displays </li></ul><ul><li>Locomotion Interfaces </li></ul><ul><li>Olfactory Displays </li></ul><ul><li>Taste Displays ? </li></ul><ul><li>Sixth Sense .. </li></ul>Extending sensory modalities
  52. 56. Sense of Balance <ul><li>Galvanic Vestibular Stimulation </li></ul>Masahiko Inami (UEC)
  53. 57. Electric Cutaneous Display Skin Nerve Electrode Dr. H. Kajimoto, U Tokyo Masahiko Inami (UEC)
  54. 58. Smart Touch Masahiko Inami (UEC)
  55. 59. Scent Display Yasuyuki Yanagida, ATR Spatio-Temporal Controlled Delivery of Scent
  56. 60. Air Cannon Generating a doughnut-shaped vortex ring Aperture Push
  57. 61. Air Cannon
  58. 62. Prototype System <ul><li>Nose tracking </li></ul><ul><li>Multiple scents </li></ul><ul><li>Scent switching </li></ul><ul><li>Improved delivery range </li></ul>Yasuyuki Yanagida, ATR
  59. 63. Beyond White Objects Grossberg, Nayar et al
  60. 65. Grossberg, Nayar et al
  61. 66. The Virtual Showcase Oliver Bimber Tracking and Mirrors
  62. 67. Displays Contenders Organic LED Light Emitting Polymers E-Ink
  63. 68. Holodust Perlin et al
  64. 69. Fake World
  65. 70. Beyond Gouraud Shading of White Objects <ul><li>Participating Media </li></ul><ul><ul><li>simulating or in presence of smoke, fog </li></ul></ul><ul><li>Complex Geometry </li></ul><ul><ul><li>Spaghetti </li></ul></ul><ul><li>Complex Reflectance </li></ul><ul><ul><li>Arbitrary appearance on arbitrary BRDF surfaces </li></ul></ul><ul><li>Shape and Motion </li></ul><ul><ul><li>Transformation and Animation of real surfaces </li></ul></ul><ul><li>NPR, Distortions, Perceptual factors </li></ul><ul><ul><li>Great thesis topics .. </li></ul></ul>
  66. 71. Recap <ul><li>Special Effects in Real World </li></ul><ul><ul><li>Photorealism yet to impact daily life </li></ul></ul><ul><ul><li>Poor Man’s Palace </li></ul></ul><ul><li>Spatial Augmented Reality </li></ul><ul><ul><li>Solution for fusion </li></ul></ul><ul><ul><li>Geometry, Photometry, Id </li></ul></ul><ul><ul><li>Sense, Control, Compensate </li></ul></ul><ul><ul><li>Needs hardware components: Projectors, RFID, Sensors </li></ul></ul><ul><li>Open Problems </li></ul><ul><ul><li>Natural phenomenon, complex BRDF, other displays </li></ul></ul><ul><ul><li>All senses: haptic, olfactory, preprioception </li></ul></ul><ul><li>Next Challenge: Photorealism </li></ul>
  67. 72. Spatial Augmented Reality <ul><li>Graphics </li></ul><ul><ul><li>Photorealism yet to impact daily life </li></ul></ul><ul><ul><li>Special effects in real world </li></ul></ul><ul><li>Spatial Augmentation </li></ul><ul><ul><li>Solution for fusion </li></ul></ul><ul><ul><li>Needs hardware components </li></ul></ul><ul><ul><li>Projectors, RFID, Sensors </li></ul></ul><ul><li>Next Challenge: Photorealism </li></ul>raskar.com
  68. 73. More <ul><li>Networked fake world </li></ul><ul><li>Poor man’s palace </li></ul><ul><li>Intelligence moving from PC to Proj to Surf </li></ul><ul><li>Electronic items eventually become cheaper than those scavenged from nature </li></ul>
  69. 74. Special Effects <ul><ul><li>Photorealism </li></ul></ul><ul><ul><ul><li>Limited to display screens </li></ul></ul></ul><ul><ul><ul><li>Does not affect daily life in real time </li></ul></ul></ul><ul><ul><ul><ul><li>Entertainment and Visualization </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Unlike other fields of CS </li></ul></ul></ul></ul><ul><ul><ul><li>Provoke not offend </li></ul></ul></ul><ul><ul><ul><li>The primary advantage AR has over VR is that AR uses the parts of the body that aren't just the eyes and ears: proprioception, vestibular perception, and othe cues that old-fashioned VR just can't handle. The disjunct between vestibular information and visual information that you get in VR is the source of the motion-sickness that often accompanies it. </li></ul></ul></ul><ul><ul><ul><li>VR, like a lot of early 'cyberspace' mythology, was built on an unrealistic rejection of the body, and a fantasy of &quot;pure mind.&quot; </li></ul></ul></ul><ul><ul><ul><li>Bud Fake Cup of Coffee 3000 </li></ul></ul></ul><ul><ul><ul><li>http://www.visit4info.com/details.cfm?adid=21038 </li></ul></ul></ul>
  70. 75. Applications <ul><ul><li>Static setup </li></ul></ul><ul><ul><ul><li>Color everyday objects </li></ul></ul></ul><ul><ul><ul><li>Engineering drilling locations </li></ul></ul></ul><ul><ul><ul><li>Stage backdrops, seasons or aging </li></ul></ul></ul><ul><ul><li>Laser shows on buildings </li></ul></ul><ul><ul><li>Simulate motion </li></ul></ul><ul><ul><ul><li>Car shows, wheels, dirt marks </li></ul></ul></ul><ul><ul><ul><li>Virtual moving lights </li></ul></ul></ul>
  71. 76. Applications Indoors, under controlled lighting <ul><ul><li>Architectural models </li></ul></ul><ul><ul><ul><li>Augment walk-around scaled model of buildings </li></ul></ul></ul><ul><ul><ul><li>Project and ‘paint’ surface colors, textures </li></ul></ul></ul><ul><ul><ul><li>Lighting, sunlight, seasons </li></ul></ul></ul><ul><ul><ul><li>Internal structure, pipes, wiring </li></ul></ul></ul><ul><ul><li>Assembly line </li></ul></ul><ul><ul><ul><li>Instructional text, images and procedures </li></ul></ul></ul><ul><ul><li>Entertainment </li></ul></ul><ul><ul><ul><li>Live shows, exhibits, demonstrations </li></ul></ul></ul>
  72. 77. Tools for Augmentation <ul><li>Environment </li></ul><ul><ul><li>Sense, Compensate and Control </li></ul></ul><ul><ul><li>Id, Geometry, Photometry(reflectance/Illumination), Id </li></ul></ul><ul><li>Display </li></ul><ul><li>Rendering </li></ul>

×