Light Field Technology


Published on

These slides use concepts from my (Jeff Funk) course entitled analyzing hi-tech opportunities to analyze how Light Field Technology is becoming economic feasible for an increasing number of applications. Light Field Cameras record all of the light fields in a picture instead of just one light field. This capability enables users to change the focus of pictures after they have been taken and to more easily record 3D data. These features are becoming economically feasible improvements because of rapid improvements in camera chips and micro-lens arrays (an example of micro-electronic mechanical systems, MEMS). These features offer alternative ways to do 3D sensing for automated vehicles and augmented reality and can enable faster data collection with telescopes.

Published in: Business, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Light Field Technology

  1. 1. Light Field Technology Chen Zhao, Chen Zhi, John Allen Ray, Ung Guan Wah, Zhou Xintao For information on other technologies, please see Jeff Funk’s slide share account ( or his book with Chris Magee: Exponential Change: What drives it? What does it tell us about the future? ebook/dp/B00HPSAYEM/ref=sr_1_1?ie=UTF8&qid=1398325920&sr=8-1&keywords=exponential+change
  2. 2. What is light field technology? • Light goes through the main lens and then hit a special lens • The special lens incorporates a compound lens known as a micro-lens array (MLA) • MLA re-group sensors (e.g. CCD, CMOS) into small groups • Capture information on the direction, color, and luminosity of millions of individual rays of light • Computationally reconstruct the light field (3D) • Lytro camera Generalized conceptual diagram of light field camera
  3. 3. Driving Force Refocusing pictures Optical correction 3D scanning, modeling Price drop on image sensor and MLA
  4. 4. Timeline of Light Field Camera (LFC) 1992 Plenoptic camera proposed in research 2006 Refocus Imaging (later Lytro) 2008 Foundation of Raytrix 2010 R11 first commercialized LF camera, 30,000 euros, 3 megapixels 2011 Lytro launched, first consumer LFC, 499USD. 1.1 megapixels 2014 Jan, Toshiba started sample shipment of Dual camera module with depth data, 50 USD, 13 megapixels 2015 first half, Pelican Imaging shipment planned.
  5. 5. Rates of Improvement (LFC) 1 10 100 1000 10000 100000 2009 2010 2011 2012 2013 2014 2015 2016 ProductioncostperMegapixels(USD) R11 Lytro Toshiba Pelican 0 100000 200000 300000 400000 500000 600000 2009 2010 2011 2012 2013 2014 2015 Totalnumberofmicrolenses R11 Lytro Toshiba Year Model Cost (USD) Megapixel Pixel Size (um) Microlenses Lense size (um) 2010 R11 39,000 3 9 40,000 200 2011 Lytro 499 1 1.4 130,000 13.89 2014 Toshiba 50 13 1.4 500,000 30 ? Pelican 20 8 ? ? ?
  6. 6. T. Suzuki, “Challenges of Image-Sensor Development”, ISSCC, 2010 Changes in Scale Impacted on Cost per pixel of Camera Chips
  7. 7. Microlens Array (MLA) Modern fabrication methods: • Photolithography based on standard semiconductor processing technology • Feature size less than 1mm and often as small as 10 um Other applications: • Coupling light to optical fibres • Increase light collection efficiency of CCD arrays • Focus light in digital projectors • Concentrators for high efficiency photovoltaics
  8. 8. Applications Cameras 3D Sensing Augment RealityTelescope/Microscope Autonomous vehicles
  9. 9. Mobile Software Refocus 2014 is becoming the year of mobile software refocus Nokia Lumia Sony XperiaTM Z2 (waterproof) Samsung Galaxy S5 LG G Pro 2 Meizu MX3
  10. 10. Cameras - Reduction in size Toshiba • 18.0 mm x 12.0 mm x 4.65 mm • Dual camera module • 13 Megapixels • CMOS Light Field sensor • 500,000 microlenses (5X more than Lytro) • Module shipped in Jan 2014 • Sample US$50
  11. 11. Cameras - Reduction in size Pelican Imaging • Developed extremely thin, and cheap light field camera module • Height 3mm • Array of small cameras • Production cost US$20 • 8 Megapixels output • Future Nokia Lumia confirmed with this feature Nleclfgqn_U
  12. 12. Applications Cameras 3D Sensing Augment RealityTelescope/Microscope Autonomous vehicles
  13. 13. A light-field picture contains more information about depth than simply correspondence which allow us to capture the real world in unparalleled detail (4D). Rendering complexity is independent of scene complexity. Processing speed is fast No need to worry about the focus of the scanner lenses. Why light field for 3D sensing is better
  14. 14. 1. Stereoscopic vision: Currently most common 3D sensor approach Passive range determination via stereoscopic vision utilizes the disparity in viewpoints between a pair of near-identical cameras to measure the distance to a subject of interest. 3D sensing Technology
  15. 15. 2. Structured light: Replaces the stereoscopic vision sensor's second imaging sensor with a projection component. Similar to stereoscopic vision techniques, this approach takes advantage of the known camera-to-projector separation to locate a specific point between them and compute the depth with triangulation algorithms. 3D sensing Technology
  16. 16. 3. Time of flight An indirect system to obtain travel time information by measuring the delay or phase shift of a modulated optical signal for all pixels in the scene. The ToF sensor in the system consists of an array of pixels, where each pixel is capable of determining the distance to the scene. 3D sensing Technology
  17. 17. Stereoscopic vision Structured light Time of flight(TOF) Light Field Software complexity High High Low Low Material Cost Low High/Middle Middle Low Response time Middle Slow Fast Fast Low light Weak Light source dep (IR or visible) Good (IR, laser) Good Outdoor Good Weak Fair Good Depth (“z”) accuracy cm μm ~ cm mm ~ cm μm ~ cm Range Mid range Very short range(cm) to mid range (4- 6m) Short range(<1m) to long range(~40m) Very short range(cm) to long rang (~100m) Application Device control 3D movie 3D scanning
  18. 18. 3D-vision gesture control system, which is a highly precise and reliable user interface for interacting with any display screen from any distance. Whether on a personal computer, set top box, television set, mobile device, game console, digital sign or interactive kiosk, The depth tracking software enables users to control onscreen interaction with simple hand motions instead of a remote control, keyboard or touch screen. 3D sensing control
  19. 19. 3D Dental Scanner
  20. 20. 3D scanning for arch
  21. 21. Interactive Billboard Interactive mirror Future Retail Industry
  22. 22. Mechanical Parts Scanning For Lockheed Martin, 3D scanner assures the right fit the first time Car scanning
  23. 23. Applications Cameras 3D Sensing Augment Reality Telescope/Microscope Autonomous vehicles
  24. 24. Is this real? Created from Light field AR Created from Hologram
  25. 25. layer_embedded&v=pky822zG4hM • Without the benefit of clear natural sight, such advances light field in AR are extraordinarily helpful • Ability to see more - inside of a patient (Diagnosis and Therapy) • Limited – type of displays are cumbersome • New equipment such as transparent screens - Displaying information and graphics about the person's condition, - combination of visualization and location tracking technology, • Medical AR technology compared with light field, ultrasound and location technology
  26. 26. AR using Light Field 3D MOULD • More details • Bring it every where you can
  27. 27. AR Tool kit (VRML) vs. Rendering Light field Reference:
  28. 28. AR Tool kit (VRML) vs. Rendering Light field Detail Processing • Constant Response • Shorter Response Reference:
  29. 29. Higher processing requirement Traditional ARNearing peak of inflated expectation Break away from Monitor & display Level 0 - Physical World Hyper Linkin Level 1 - Marker Based AR Level 2 - Markerless AR Level 3 - Augmented Vision
  30. 30. Distribution of AR Application on mobile • Increase demand in mobile device application • Install with Light Field Camera for fast response • “Shot and Focus” later
  31. 31. Representation Hologram Vs Light Field a) Depicts the representation of a hologram. b) and c) show two different representations of a light field. Reference: Eurographics 2007/D.Cohen-Or and P.Slavik, A Bidirectional Light Field-Hologram Transform Volume 26 (2007), Number 3
  32. 32. Light Field Mapping • The possibility to transform a light field into a holographic representation and vice versa. • The holographic data representation is similar to a light field • “M” transforms the light field into a holographic representation. • Method to extract depth from the input light field. • If accurate depth information is available for the light field it can optionally be added to the input Reference: Eurographics 2007/D.Cohen-Or and P.Slavik, A Bidirectional Light Field-Hologram Transform Volume 26 (2007), Number 3
  33. 33. Light Field Mapping • Depth Reconstruction from Light Fields • Effects of Loss of Data from Hologram Reference: Eurographics 2007/D.Cohen-Or and P.Slavik, A Bidirectional Light Field-Hologram Transform Volume 26 (2007), Number 3
  34. 34. Hologram Vs Light Field Reference: Eurographics 2007/D.Cohen-Or and P.Slavik, A Bidirectional Light Field-Hologram Transform Volume 26 (2007), Number 3
  35. 35. Hologram Vs Light Field • Hologram - an illustration of direct output of holographic content on future generation holographic displays. • Light field - far more efficient for conventional 2D frame buffer displays. • Light Field - the versatility and the power of transformation on synthetic light fields, real light fields and digitally recorded holograms. • The rendered images can be evaluated directly from the holographic representation or through light field rendering. • Light field capable of simulating different aperture sizes as well as focal length – Versatile displays Future • Holograms can be captured using a light field camera • Take advantage of the realism and detail preserving benefits of a real light field while giving the possibility of a 3D output on a holographic screen
  36. 36. AR Light Field Possible Improvements • See-through displays • New tracking sensors • Interfaces and Interactions
  37. 37. Applications Cameras 3D Sensing Augment Reality Telescope/M icroscope Autonomous vehicles
  38. 38. Scientific Applications Light-Field Telescope Light-Field Microscope
  39. 39. Light-Field Telescope Source: JonathanWedd, Jan van der Laan, Eric Lavelle, and David Stoker. “A High-Magnification Light-Field Telescope forExtended Depth-of-Field Biometric Imaging”
  40. 40. Light-Field Telescope  Take the image faster and refocus the image after taking  Increase magnification  Achieve large depth of field and high lateral image resolution simultaneously  Capture different wavelengths
  41. 41. Light Field Microscope  A compact Light Field Microscope Designed by Stanford  Consists of an ordinary research microscope and cooled scientific camera  A microlens array is inserted Source: Marc Levoy, Ren Ng, Andrew Adams, Matthew Footer, Mark Horowitz. “Light Field Microscopy”. Stanford University
  42. 42. Light Field Microscope Source: Marc Levoy, Ren Ng, Andrew Adams, Matthew Footer, Mark Horowitz. “Light Field Microscopy”. Stanford University
  43. 43. Light Field Microscope  Captures light fields of biological specimens in a single snapshot  Offers 3D functional imaging of neuronal activity in entire organisms at single cell level  Separates image acquisition from the selection of viewpoint and focus  Captures video of high speed moving specimens
  44. 44. Applications Cameras 3D Sensing Augment RealityTelescope/Microscope Autonomous vehicles
  45. 45. Autonomous Vehicle Applications Consumer Passenger Vehicles Agricultural Vehicles Military Combat Vehicles Logistics/Supplies Search & Rescue Vehicles M.Bellone, et al, “Unevenness Point Descriptor for Terrain Analysis in Mobile Robot Applications”, 2012
  46. 46. LFC Vs. Laser Based Systems Advantages: Easier data interpretation Significantly Lower Cost Lower Power Requirement Comparable performance Disadvantages: Smaller field of view (multiple cameras required) D. Stavens, “LEARNING TO DRIVE: PERCEPTION FOR AUTONOMOUS CARS”, 2011
  47. 47. Light Field Depth Map M.Tao, et al, “Depth from Combining Defocus and Correspondence Using Light-Field Cameras”, 2013
  48. 48. Light Field Terrain Analysis M.Bellone, et al, “Unevenness Point Descriptor for Terrain Analysis in Mobile Robot Applications”, 2012
  49. 49. Light Field Terrain Analysis M.Bellone, et al, “Unevenness Point Descriptor for Terrain Analysis in Mobile Robot Applications”, 2012
  50. 50. Future of Light Field We have analyzed advantages of LF We showed applications of LF in cameras, mobile phones, 3D scanning, AR, scientific research, and autonomous vehicles Future applications of LF will enable us to do more simple and fast 3D applications