• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
 

Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School

on

  • 1,740 views

How to do research, Idea Hexagon, Rank and Sparsity in imaging problems, Looking around corners, compressive sensing of periodic phenomena, 3D displays, fast computation

How to do research, Idea Hexagon, Rank and Sparsity in imaging problems, Looking around corners, compressive sensing of periodic phenomena, 3D displays, fast computation

Statistics

Views

Total Views
1,740
Views on SlideShare
1,739
Embed Views
1

Actions

Likes
0
Downloads
62
Comments
1

1 Embed 1

http://ubcc.ap01.aws.af.cm 1

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Six ways of coming up with new ideas based on an idea ‘X’.Ramesh RaskarAssociate ProfessorMIT Media Labhttp://raskar.infohttp://cameraculture.infoFull Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010http://raskar.infohttp://cameraculture.info
  • Full Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
  • X up: Airbags for car, for helicopter
  • Full Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
  • http://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_Catechism
  • Five on Five= If more than five teams in the world are doing the same research, don’t do it.= If you disappear for five years, will someone do it anyway? Then your idea is not that great anyway. = Can you explain your work in five sentences to your grandmahow it will impact human life?= If you can explain the idea in five minutes to a student and disappear for five years, will s/he be able to do it on her/his own without additional input from you/without iterations .. It is too obvious and lacks depth .. Don’t do it.= Strive to work on ideas that may require five+ disciplines .. Today’s research is highly team-driven and more diverse the required team composition, more fun you will have and also indicates a natural barrier to entry for others satisfying condition 1 and 2Much like the food pyramid, five servings are the goal and will make you stronger .. But ok if your research project does not satisfy all five conditions
  • But the world is 4D
  • See computationalphotography.orgMove away from obsession about SNR, space-bandwidth, diffraction limit and so on
  • My work involves creative new ways to play with light by co-designing optical and digital processing.My work lies at the INTERSECTION of processing of photons and processing of bits.At MERL, I transformed the field of computational photography, with key papers and impact on productsAt Media Lab, I invented a new field ‘computational light transport’
  • Compressive sensing via random projections not suitable for images and even videos
  • Rudy Burger, ‘don’t use flash and destroy the image’Can we use flash not just for improving scene brightness but for enhancing the mood? Like in studio lights?Main difference between professionals and consumers is lighting.
  • http://cameraculture.infohttp://raskar.info
  • My idea is to use the multiple bounces of light i.e. echoes of light.Echoes of sound -> Echoes of lightWe all know about echoes of sound.But sounds travels slow and we can actually hear the echoesLight travels fast so we need specialized hardware to ‘listen’ to these echoes.So we end up using light sources and cameras that run at a trillion frames per second (not a million and not a billion, but trillion)
  • Trillion fps camera (which was previously used only for specialized biochemistry expt)This new form of imaging is possible by fusion of dissimilar .. A specialized camera previously used only in biochemistry labs and a new computational method that analyzes multiple bounces of light.I started the project just before I joined MIT in summer 2008.The hardware we use is in the lab of Prof Bawendi, MIT Chemistry, who is now a collaborator
  • Here is a road map for this ambitious research project based on time-resolved imaging .. Non line of sight Looking around corner (LaC) is just one example .. Such Time resolved imaging requires one to develop a completely new set of tool for understanding our world.This is a project I started just before coming to MIT via an NSF proposal.
  • The reconstruction is very low right now, about 80x80 pixels. So these are just baby steps.
  • The reconstruction is very low right now, about 80x80 pixels. So these are just baby steps.Data collected and reconstructions program by Andreas Velten, scientist in my group
  • Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011
  • A cross section through a single M. rhetenor scale. Light reflected off each level of the “Christmas tree structure” gives the butterfly its iridescent color. Credit: Pete Vukusic, University of Exeter
  • Lanman, Hirsch, Kim, RaskarSiggraph Asia 2010

Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School Presentation Transcript

  • Computational Light Transport
    and
    Computational Photography:
    Inverse problems
    Camera Culture
    Ramesh Raskar
    Ramesh Raskar
    http://raskar.info
    MIT Media Lab
    raskar@mit.edu
  • How to Invent?
    After X, what is neXt
    Full Presentation at
    http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
    Ramesh Raskar, MIT Media Lab
  • Ramesh Raskar, http://raskar.info
    X+Y
    X
    neXt
    Xd
    X
    X++
    X
    Full Presentation at
    http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
  • Simple Exercise ..
    Image Compression
    Save Bandwidth and storage
    What is neXt
  • Strategy #1: Xd
    Extend it to next (or some other) dimension ..
  • X =
    Idea you just heard
    Concept
    Patent
    New Product/Best project/invention award
    Product feature
    Design
    Art
    Algorithm
  • Ramesh Raskar, http://raskar.info
    X+Y
    X
    neXt
    Xd
    X
    X++
    X
    Full Presentation at
    http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
  • Research ..
    http://raskar.info
    How to come up w ideas: Idea Hexagon
    How to write a paper
    How to give a talk
    Open research problems
    How to decide merit of a project
    How to attend a conference, brainstorm
    Facebook.com/ rRaskar
    Tips
    Get on Seminar/Talks mailing lists worldwide
    http://www.cs.virginia.edu/~robins/YouAndYourResearch.html
    Why do so few scientists make significant contributions and so many are forgotten in the long run?
    Highly recommended Hamming talk at Bell Labs
  • Is project worthwhile? Heilmeier's Questions
    http://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_Catechism
    What
    What are you trying to do? Articulate your objectives using absolutely no jargon.
    Related work
    How is it done today, and what are the limits of current practice?
    Contribution
    What's new in your approach and why do you think it will be successful?
    Motivation
    Who cares?
    If you're successful, what difference will it make?
    Challenges
    What are the risks and the payoffs?
    How much will it cost?
    How long will it take?
    Evaluation
    What are the midterm and final "exams" to check for success?
    Raskar additions:
    Why now? (why not before, what’s new that makes possible)
    Why us? (wrong answers: I am smart, I can work harder than others)
  • Great Research: Strive for Five
    Before Five teams
    Be first, often let others do details
    Beyond Five years
    What no one is thinking about
    Within Five layers of ‘Human’ Impact
    Relevance
    Beyond Five minutes of description
    Deep, iterative, participatory
    Fusing Five+ Expertise
    Multi-disciplinary, proactive
    Ramesh Raskar, http://raskar.info
  • MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar
    Inverse Problems
    How to do Research in Imaging
    • Inverse Problems, Reconstruction, Rank and Sparsity
    Co-design of Optics and Computation
    Photons not just pixels
    Mid-level cues
    Computational Photography
    Open research problems
    Compressive Sensing for High Speed Events
    Limits of CS for general imaging
    Computational Light Transport
    Looking Around Corners, trillion fps
    Lightfields: 3D Displays and Holograms
  • Tools
    for
    Visual Computing
    Shadow
    Refractive
    Reflective
    Fernald, Science [Sept 2006]
  • Computational Photography
    Camera Culture
    Ramesh Raskar
  • Traditional Photography
    Detector
    Lens
    Pixels
    Mimics Human Eye for a Single Snapshot:
    Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static world
    Image
    Courtesy: Shree Nayar
  • Picture
    Computational Camera + Photography: Optics, Sensors and Computations
    GeneralizedSensor
    Generalized
    Optics
    Computations
    Ray Reconstruction
    4D Ray Bender
    Upto 4D Ray Sampler
    Merged Views, Programmable focus and dynamic range, Closed-loop Controlled Illumination, Coded exposure/apertures
  • Computational Photography
    Novel Illumination
    Light Sources
    Modulators
    Computational Cameras
    Generalized
    Optics
    GeneralizedSensor
    Generalized
    Optics
    Processing
    4D Incident Lighting
    4D Ray Bender
    Ray Reconstruction
    Upto 4D Ray Sampler
    4D Light Field
    Display
    Scene: 8D Ray Modulator
    Recreate 4D Lightfield
  • Computational Photography [Raskar and Tumblin]
    captures a machine-readable representation of our world to
    hyper-realistically synthesize the essence of our visual experience.
    Resources
    ICCP 2012, Seattle Apr 2012
    Papers due Dec 2nd, 2011
    http://wikipedia.org/computational_photography
    http://raskar.info/photo
  • Computational Photography
    Computational Photography aims to make progress on both axis
    Phototourism
    Comprehensive
    Essence
    Scene completion from photos
    Augmented Human Experience
    Looking Around Corners
    Priors
    Capture
    Human Stereo Vision
    Metadata
    Coded
    Depth
    fg/bg
    Non-visual Data, GPS
    Virtual Object Insertion
    Spectrum
    Decomposition problems
    8D reflectance field
    Direct/Global
    LightFields
    Relighting
    Epsilon
    Angle, spectrum aware
    Camera Array
    HDR, FoV
    Focal stack
    Resolution
    Material editing from single photo
    Digital
    Motion Magnification
    Raw
    Low Level
    Mid Level
    HighLevel
    Hyper realism
    Synthesis/Analysis
  • Co-designing Optical and Digital Processing
    Computational Light Transport
    Optics
    Displays
    Sensors
    Computational Photography
    Photon Hacking
    Illumination
    Signal Processing
    Computer Vision
    Machine Learning
    Bit Hacking
  • Take home points
    Co-design of hw/sw
    Avoid computational or optical chauvinism in imaging
    (Camera flash/Kinect)
    Hardware cost going to zero, Parallel technology trends
    Computer vision not just mimicking human vision/perception
    Borrow ideas from other fields: astronomy, scientific imaging, audio, communications
    Photons not just Pixels
    Change the rules of the game
    Optics, Sensors, Illum,
    Priors, Sparsity, Transforms
    Meta-data, Internet collection, Crowdsourcing
  • Computational Photography
    Wish List:
    Open Research Problems
    Camera Culture
    Ramesh Raskar
  • Wish #1
    Ultimate Post-capture Control
    Camera Culture
    Ramesh Raskar
  • Digital Refocusing using Light Field Camera
    125μ square-sided microlenses
    [Ng et al 2005]
  • Motion Blur in Low Light
  • Traditional
    Blurred Photo
    Deblurred Image
  • Fluttered Shutter Camera
    Raskar, Agrawal, Tumblin Siggraph2006
    Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
  • Preserves High Spatial Frequencies
    Fourier Transform
    Sharp Photo
    Blurred Photo
    PSF == Broadband Function
    Flutter Shutter: Shutter is OPEN and CLOSED
  • Coded Exposure
    Traditional
    Deblurred Image
    Deblurred Image
    Image of Static Object
  • Motion Blur in Low Light
  • Fast periodic phenomena
    Vocal folds flapping at 40.4 Hz
    Bottling line
    4000 fps hi-speed camera
    500 fps hi-speed camera
  • Compressive Sensing
    Single Pixel Camera
    image
    compressive image
    measurement matrix
  • Periodic signals
    -fP
    -2fP
    -4fP
    3fP
    -3fP
    0
    fMax
    - fMax
    2fP
    fP=1/P
    4fP
    Periodic signal x(t) with period P
    t
    P = 16ms
    Periodic signal with period P and band-limited to fMax = 500 Hz.
    Fourier transform is non-zero only at multiples of fP=1/P ~ 63Hz.
  • High speed camera
    P = 16ms
    Ts = 1/(2 fMax)
    -fP
    -2fP
    -4fP
    -3fP
    4fP
    3fP
    2fP
    0
    fMax
    - fMax
    fP=1/P
    Nyquist Sampling of x(t)
    Periodic signal has regularly spaced, sparse Fourier coefficients.
    Is it necessary to use a high-speed video camera? Why waste bandwidth?
  • Traditional Strobing
    Use low frame-rate camera and generate beat frequencies.
    P
    t
    Low exposure to avoid blurring. Low light throughput.
    Period known apriori.
    Strobing animation credit Wikipedia
  • t
    P
    Random Projections Per Frame of Camera using Coded Strobing Photography
    In every exposure duration observe different linear combinations of the periodic signal.
    Advantage of the design
    • Exposure coding independent of the frequency
    • On an average, light throughput is 50%
    Coded Strobing Photography. Reddy, D., Veeraraghavan, A., Raskar, R. IEEE PAMI 2011
  • Observation Model
    x at 2000fps
    y at 25fps
  • Signal Model
    x at 2000fps
    y at 25fps
  • Signal & Observation Model
    Ais M x N, M<<N
    x at 2000fps
    y at 25fps
    N / M = 2000 / 25 = 80
  • Recovery: Sparsity
    Very few non-zero elements
    y = A s
    Observed values
    Mixing matrix
    Structured Sparse Coefficients
    Basis Pursuit De-noising
  • Simulation on hi-speed toothbrush
    25fps normal camera
    25fps coded strobing camera
    Reconstructed frames
    2000fps hi-speed camera
    ~100X speedup
  • Rotating mill tool
    Mill tool rotating at 50Hz
    Reconstructed Video at 2000fps
    Normal Video: 25fps
    Coded Strobing Video: 25fps
    Blur increases as rotational velocity increases
    rotating at 200Hz
    rotating at 150Hz
    rotating at 100Hz
    increasing blur
  • Compressive Sensing for Images .. A good idea?
    Single Pixel Camera
    image
    compressive image
    measurement matrix
  • Is Randomized Projection-based Captureapt for Natural Images ?
    Periodic Signals
    Progressive Projections
    Randomized Projections
    Compression Ratio
    [Pandharkar, Veeraraghavan, Raskar 2009]
  • Compact ProgrammableLights ?
  • Wish #1
    Ultimate Post-capture Control
    • Digital Refocus and Motion blur
    • Emulate studio light from compact flash
    Camera Culture
    Ramesh Raskar
  • Wish #2
    Freedom from Form
    • Size, Weight, Power, UI
    • Flat camera:
    Bidirectional screen (BiDi)
    • Shallow DoF from tiny lens
    Camera Culture
    Ramesh Raskar
  • Wish #3
    Understand the World
    Camera Culture
    Ramesh Raskar
  • Convert single 2D photo into 3D ?
    Snavely, Seitz, Szeliski
    U of Washington/Microsoft: Photosynth
  • Exploit Community Photo Collections
    U of Washington/Microsoft: Photosynth
  • Wish #3
    Understand the World
    • Identify/recognize Materials
    • 3D Awareness
    • Interact with information
    Camera Culture
    Ramesh Raskar
  • Wish #4
    Sharing Visual Experience
    • LifeLog Auto-summary
    • Privacy in public and authentication
    • Hyper-real Photo Frames
    • Print ‘material’
    Camera Culture
    Ramesh Raskar
  • Wish #5
    Capturing Essence
    Camera Culture
    Ramesh Raskar
  • What are the problems with ‘real’ photo in conveying information ?
    Why do we hire artists to draw what can be photographed ?
  • Shadows
    Clutter
    Many Colors
    Highlight Shape Edges
    Mark moving parts
    Basic colors
  • Depth Edges with MultiFlash
    Raskar, Tan, Feris, Jingyi Yu, Turk – ACM SIGGRAPH 2004
  • Depth Discontinuities
    Internal and externalShape boundaries, Occluding contour, Silhouettes
  • Depth Edges
  • Our Method
    Canny
  • Result
    Photo
    Canny Intensity Edge Detection
    Our Method
  • Questions
    What will a camera look like in 10,20 years?
    How will a billion networked and portable cameras change the social culture?
    How will online photo collections transform visual social computing?
    How will movie making/new reporting change?
  • Photos of tomorrow: computed not recorded
    http://scalarmotion.wordpress.com/2009/03/15/propeller-image-aliasing/
  • Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info
    Sensor
    Computational Photography Wish List
    • Post-capture control
    • Emulate studio lights with compact flash
    • Focus and motion blur
    • New forms
    • Flat camera, large LCDs as cameras
    • Image destabilization for larger aperture
    • Understand the world
    • Real or fake
    • Place 2D photo into 3D
    • Look around corner
    • Bokode: long distance barcode
    • Sharing
    • Lifelogs auto summary
    • Privacy/Verification
    • 6D photoframes
    • Essence
    • New visual arts
    • Multi-flash camera
    • Delta-camera and Blind-camera
  • Take home points
    Co-design of hw/sw
    Avoid computational or optical chauvinism in imaging
    (Camera flash/Kinect)
    Hardware cost going to zero, Parallel technology trends
    Computer vision not just mimicking human vision/perception
    Borrow ideas from other fields: astronomy, scientific imaging, audio, communications
    Photons not just Pixels
    Change the rules of the game
    Optics, Sensors, Illum,
    Priors, Sparsity, Transforms
    Meta-data, Internet collection, Crowdsourcing
  • MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar
    Inverse Problems
    How to do Research in Imaging
    • Inverse Problems, Reconstruction, Rank and Sparsity
    Co-design of Optics and Computation
    Photons not just pixels
    Mid-level cues
    Computational Photography
    Open research problems
    Compressive Sensing for High Speed Events
    Limits of CS for general imaging
    Computational Light Transport
    Looking Around Corners, trillion fps
    Lightfields: 3D Displays and Holograms
  • Every Photon has a Story
  • What isaround the corner ?
  • Can you look around the corner ?
  • Multi-path Analysis
    2nd Bounce
    1st Bounce
    3rd Bounce
  • Femto-Photography (Transient Imaging)
    FemtoFlash
    Trillion FPS camera
    With M Bawendi, MIT Chemistry
    Serious Sync
    Computational Optics
    • 2011: CVPR (Pandharkar, Velten, Bardagjy, Bawendi, Raskar)
    • 2009: Marr PrizeHonorable Mention (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)
    • 2008: Transient Light Transport (Raskar, Davis, March 2008)
  • Inverting Light Transport
    Direct/Global
    Multiple Scattering
    [Seitz , Kutulakos, Matsushita 2005]
    [Nayar, Raskar et al 2006]
    [Atcheson et al 2008]
    [Kutulakos, Steger 2005]
    Dual Photography
    LIDAR
    [Sen et al 2005]
  • Multi-Dimensional Light Transport
    5-D Transport
    Gigapan
  • Collision avoidance, robot navigation, …
  • z
    x
    S
    L
    R
    s
    Occluder
    Streak-camera
    3rd bounce
    C
    Laser beam
    B
    Echoes of Light
  • Steady State 4D
    Impulse Response, 5D
  • Scene with
    Ultra fast illumination and camera
    hidden elements
    Raw
    5D Capture
    Time profiles
    Signal
    Proc.
    Photo, geometry, reflectance beyond line of sight
    Novel light transport models and inference
    algorithms
    ®
    t
    3D Time images
    Femto-PhotographyTime Resolved Multi-path Imaging
  • Team
    Moungi G. Bawendi, Professor, Dept of Chemistry, MITJames Davis, UC Santa CruzAndreas Velten, Postdoctoral Associate, MIT Media LabRohitPandharkar, RA, MIT Media Lab
    Otkrist Gupta, RA, MIT Media LabAndrew Matthew Bardagjy, RA, MIT Media Lab
    Nikhil Naik, RA, MIT Media LabTyler Hutchison, RA, MIT Media LabEverett Lawson, MIT Media Lab
    Ramesh Raskar, Asso. Prof., MIT Media Lab
    Camera Culture
    Ramesh Raskar
  • Photos from Streak Camera
    Capture Setup
    Hidden Scene
  • Photos from Streak Camera
    Capture Setup
    Hidden Scene
    Overlay
    Reconstruction
  • Motion beyond line of sight
    Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011
  • …, bronchoscopies, …
    Participating Media
  • Photo
    First Bounce
    Later Bounces
    +
    Direct
    Global
    [Nayar, Krishnan, Grossberg, Raskar 2006]
  • Each frame = ~2ps = 0.6 mm of Light Travel
  • Ripples of Waves
  • MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar
    Inverse Problems
    How to do Research in Imaging
    • Inverse Problems, Reconstruction, Rank and Sparsity
    Co-design of Optics and Computation
    Photons not just pixels
    Mid-level cues
    Computational Photography
    Open research problems
    Compressive Sensing for High Speed Events
    Limits of CS for general imaging
    Computational Light Transport
    Looking Around Corners, trillion fps
    Lightfields: 3D Displays and Holograms
  • View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale
  • Two Layer Displays
    barrier
    lenslet
    sensor/display
    sensor/display
    PB = dim displays
    Lenslets = fixed spatial and angular resolution
    Dynamic Masks = Brighter, High spatial resolution
  • Limitations of 3D Display
    Parallaxbarrier
    LCD display
    Front
    Back
    Lanman, Hirsch, Kim, RaskarSiggraph Asia 2010
  • Light Field Analysis of Barriers
    k
    L[i,k]
    i
    `
    k
    g[k]
    i
    L[i,k]
    f[i]
    light box
  • Content-Adaptive Parallax Barriers
    L[i,k]
    `
    k
    g[k]
    i
    f[i]
    light box
  • Implementation
    Components
    • 22 inch ViewSonic FuHzion VX2265wm LCD [1680×1050 @ 120 fps]
  • Content-Adaptive Parallax Barriers
    L[i,k]
    `
    k
    g[k]
    i
    f[i]
    light box
  • Content-Adaptive Parallax Barriers
    `
    =
  • Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010
    Rank-Constrained Displays and LF Adaptation
    `
    Content-Adaptive Parallax Barriers
    =
    All dual layer display = rank-1 constraint
    Light field display is a matrix approximation problem
    Exploit content-adaptive parallax barriers
  • Optimization: Iteration 1
    rear mask: f1[i,j]
    front mask: g1[k,l]
    reconstruction (central view)
    Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
    Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 10
    rear mask: f1[i,j]
    front mask: g1[k,l]
    reconstruction (central view)
    Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
    Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 20
    rear mask: f1[i,j]
    front mask: g1[k,l]
    reconstruction (central view)
    Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
    Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 30
    rear mask: f1[i,j]
    front mask: g1[k,l]
    reconstruction (central view)
    Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
    Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 40
    rear mask: f1[i,j]
    front mask: g1[k,l]
    reconstruction (central view)
    Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
    Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 50
    rear mask: f1[i,j]
    front mask: g1[k,l]
    reconstruction (central view)
    Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
    Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 60
    rear mask: f1[i,j]
    front mask: g1[k,l]
    reconstruction (central view)
    Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
    Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 70
    rear mask: f1[i,j]
    front mask: g1[k,l]
    reconstruction (central view)
    Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
    Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 80
    rear mask: f1[i,j]
    front mask: g1[k,l]
    reconstruction (central view)
    Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
    Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 90
    rear mask: f1[i,j]
    front mask: g1[k,l]
    reconstruction (central view)
    Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
    Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
  • Content-Adaptive Front Mask (1 of 9)
  • Content-Adaptive Rear Mask (1 of 9)
  • Emitted 4D Light Field
  • Conclusion
    `
    Content-Adaptive Parallax Barriers
    =
    • Described a rank constraint for all dual-layer displays
    • With a fixed pair of masks, emitted light field is rank-1
    • Achieved higher-rank approximation using temporal multiplexing
    • With T time-multiplexed masks, emitted light field is rank-T
    • Constructed a prototype using off-the-shelf panels
    • Demonstrated light field display is a matrix approximation problem
    • Introduced content-adaptive parallax barriers
    • Applied weighted NMF to optimize weighted Euclidean distance to target
    Adaptation increases brightness and refresh rate of dual-stacked LCDs
  • Parallax Barrier: Np=103 pix.
    Hologram: NH=105 pix.
    ϕP∝w/d
    ϕH∝λ/tH
    θp=10 pix
    θH =1000 pix
    Fourier Patch
    xH =100 patches
    w
    xp=100slits
    Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009
  • Augmented Light Field
    118
    wave optics based
    rigorous but cumbersome
    Wigner Distribution Function
    WDF
    Augmented LF
    Traditional Light Field
    Traditional Light Field
    ray optics based
    simple and powerful
    Interference & Diffraction
    Interaction w/ optical elements
    Oh, Raskar, Barbastathis 2009: Augmented Light Field
  • position
    light field transformer
    LF
    LF
    LF
    LF
    (diffractive)
    optical element
    Reference
    plane
    LF propagation
    LF propagation
    Light Fields
    Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters
    angle
  • Augmented Lightfield for Wave Optics Effects
    WDF
    Wigner Distribution Function
    Augmented Light Field
    Light Field
    LF
    LF < WDF
    Lacks phase properties
    Ignores diffraction, interferrence
    Radiance = Positive
    ALF ~ WDF
    Supports coherent/incoherent
    Radiance = Positive/Negative
    Virtual light sources
  • Free-space propagation
    Light field transformer
    Virtual light projector
    Possibly negative radiance
    121
  • Lightfieldvs Hologram Displays
  • Is hologram just another ray-based light field?
    Can a hologram create any intensity distribution in 3D?
    Why hologram creates a ‘wavefront’ but PB does not?
    Why hologram creates automatic accommodation cues?
    What is the effective resolution of HG vs PB?
  • Zooming into the Light Field
    Rays: No Bending
    1 Fresnel HG Patch
    p Wm
    *
    *
    p d(θ)
    q d(θ)
    *
    q
    q
    p
    p
    *
    q Wm
    L(x,θ)
    W(x,u)
    Wm= sinc
    d = delta
    u
    θ
  • Algebraic Rank Constraint
    Rank-3
    Rank-1
    Rank-1
    s1*
    s1
    m2
    s1*
    m2
    s1
    (a) Parallax Barrier
    (c) Hybrid
    (b) Hologram
    s1
    s1
  • (a) Two Slits, Coherent
    Interference

    Rank-1
    -1
    Transform
    u
    -Transform
    R45, D
    x
    <t(x+xʹ/2)t*(x-xʹ/2)>
    t(x1)t*(x2)
    t(x+xʹ/2)t*(x-xʹ/2)
    W(x,u)
  • L1(x,θ)
    (a)
    L1
    L2(x,θ)
    L2
    L3(x,θ)
    s1
    m2
    L3
    hH
    ϕ1
    ϕ1
    z2
    ϕ1
    ϕ1
    z1
    r
    d
    L3(x,θ)
    L1(x,θ)
    L2(x,θ)
  • Is hologram just another ray-based light field?
    Can a hologram create any intensity distribution in 3D?
    Why hologram creates a ‘wavefront’ but PB does not?
    Why hologram creates automatic accommodation cues?
    What is the effective resolution of HG vs PB?
  • Three Questions
    What are the benefits of higher dimensional imaging?
    Why is the algebraic rank of a Light Field not full?
    What makes looking around the corner possible?
  • How to do Research in Imaging
    http://raskar.info
    How to come up w ideas: Idea Hexagon
    How to write a paper
    How to give a talk
    Open research problems
    How to decide merit of a project
    How to attend a conference, brainstorm
    Facebook.com/ rRaskar
    Tips
    Get on Seminar/Talks mailing lists worldwide
    http://www.cs.virginia.edu/~robins/YouAndYourResearch.html
    Why do so few scientists make significant contributions and so many are forgotten in the long run?
    Highly recommended Hamming talk at Bell Labs
  • Take home points
    Co-design of hw/sw
    Avoid computational or optical chauvinism in imaging
    (Camera flash/Kinect)
    Hardware cost going to zero, Parallel technology trends
    Computer vision not just mimicking human vision/perception
    Borrow ideas from other fields: astronomy, scientific imaging, audio, communications
    Photons not just Pixels
    Change the rules of the game
    Optics, Sensors, Illum,
    Priors, Sparsity, Transforms
    Meta-data, Internet collection, Crowdsourcing
  • MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar
    Inverse Problems
    How to do Research in Imaging
    • Inverse Problems, Reconstruction, Rank and Sparsity
    Co-design of Optics and Computation
    Photons not just pixels
    Mid-level cues
    Computational Photography
    Open research problems
    Compressive Sensing for High Speed Events
    Limits of CS for general imaging
    Computational Light Transport
    Looking Around Corners, trillion fps
    Lightfields: 3D Displays and Holograms
    Apply for internships/post-doc
    neXt