Spacetime Fusion Tutorial

1,559
-1

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,559
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Today I will be talking about an image-based-rendering algorithm that was used in this video enhancement paper from EGSR 2007.
  • Spacetime Fusion Tutorial

    1. 1. Using Photographs to Enhance Videos of a Static Scene Pravin Bhat 1 , C. Lawrence Zitnick 2 , Noah Snavely 1 , Aseem Agarwala 3 , Maneesh Agrawala 4 , Michael Cohen 1,2 , Brian Curless 1 , Sing Bing Kang 2 EGSR 2007 University of Washington 1 , Microsoft Research Redmond 2 University of California 3 , Adobe Systems 4
    2. 2. An overview of Spacetime Fusion
    3. 3. Motivation <ul><li>Low quality video </li></ul>Input Video
    4. 4. Motivation <ul><li>Low quality video </li></ul><ul><li>Reconstructed video </li></ul>Input Video Reconstructed Video
    5. 5. Motivation <ul><li>Low quality video </li></ul><ul><li>Reconstructed video </li></ul><ul><ul><li>Reconstructed from photos </li></ul></ul><ul><ul><li>Good spatial reconstruction </li></ul></ul><ul><ul><li>Bad temporal reconstruction </li></ul></ul>Input Video Reconstructed Video
    6. 6. Motivation <ul><li>Spacetime Fusion result </li></ul>Input Video Spacetime Fusion Result
    7. 7. Motivation <ul><li>Spacetime Fusion result </li></ul><ul><ul><li>Spatial properties of reconstruction </li></ul></ul><ul><ul><li>Temporal properties of input video </li></ul></ul>Input Video Spacetime Fusion Result
    8. 8. <ul><li>Define a 3D gradient field </li></ul>Spacetime Fusion
    9. 9. <ul><li>Define a 3D gradient field </li></ul><ul><ul><li>Spatial gradients from reconstruction </li></ul></ul><ul><ul><li>Temporal gradients from input video </li></ul></ul>Spacetime Fusion
    10. 10. <ul><li>Define a 3D gradient field </li></ul><ul><ul><li>Spatial gradients from reconstruction </li></ul></ul><ul><ul><li>Temporal gradients from input video </li></ul></ul><ul><ul><li>Key Idea </li></ul></ul><ul><ul><ul><li>Temporal gradients defined between motion compensated temporal neighbors </li></ul></ul></ul>Spacetime Fusion
    11. 11. <ul><li>Define a 3D gradient field </li></ul><ul><ul><li>Spatial gradients from reconstruction </li></ul></ul><ul><ul><li>Temporal gradients from input video </li></ul></ul><ul><ul><li>Key Idea </li></ul></ul><ul><ul><ul><li>Temporal gradients defined between motion compensated temporal neighbors </li></ul></ul></ul>Spacetime Fusion Video frame: t Video frame: t - 1
    12. 12. <ul><li>Define a 3D gradient field </li></ul><ul><ul><li>Spatial gradients from reconstruction </li></ul></ul><ul><ul><li>Temporal gradients from input video </li></ul></ul><ul><ul><li>Key Idea </li></ul></ul><ul><ul><ul><li>Temporal gradients defined between motion compensated temporal neighbors </li></ul></ul></ul>Spacetime Fusion Video frame: t Video frame: t - 1 G t <ul><ul><li>G t (x, y, t) = V (x, y, t) - V (x, y, t - 1) </li></ul></ul>
    13. 13. <ul><li>Define a 3D gradient field </li></ul><ul><ul><li>Spatial gradients from reconstruction </li></ul></ul><ul><ul><li>Temporal gradients from input video </li></ul></ul><ul><ul><li>Key Idea </li></ul></ul><ul><ul><ul><li>Temporal gradients defined between motion compensated temporal neighbors </li></ul></ul></ul>Spacetime Fusion Video frame: t Video frame: t - 1 G t <ul><ul><li>G t (x, y, t) = V (x, y, t) - V ( x - u , y - v , t - 1) </li></ul></ul>
    14. 14. <ul><li>Define a 3D gradient field </li></ul><ul><ul><li>Spatial gradients from reconstruction </li></ul></ul><ul><ul><li>Temporal gradients from input video </li></ul></ul><ul><ul><li>Key Idea </li></ul></ul><ul><ul><ul><li>Temporal gradients defined between motion compensated temporal neighbors </li></ul></ul></ul><ul><ul><ul><li>Increases compatibility between temporal gradients and spatial gradients </li></ul></ul></ul>Spacetime Fusion
    15. 15. <ul><li>Define a 3D gradient field </li></ul><ul><ul><li>Spatial gradients from reconstruction </li></ul></ul><ul><ul><li>Temporal gradients from input video </li></ul></ul><ul><ul><li>Key Idea </li></ul></ul><ul><ul><ul><li>Temporal gradients defined between motion compensated temporal neighbors </li></ul></ul></ul><ul><ul><ul><li>Increases compatibility between temporal gradients and spatial gradients </li></ul></ul></ul><ul><li>Integrate the 3D gradient field </li></ul>Spacetime Fusion
    16. 16. Spacetime Fusion <ul><li>Integrating the gradient field </li></ul>Solve linear system: Av = b
    17. 17. <ul><li>Integrating the gradient field </li></ul>Spacetime Fusion Solve linear system: Av = b Constraints: v x, y, t – v x-1, y, t = G x (x, y, t) v x, y, t – v x, y-1, t = G y (x, y, t) v x, y, t – v x-u, y-v, t = G t (x, y, t)
    18. 18. Applications
    19. 19. Enhanced Exposure
    20. 20. Input Video Edit Propagation
    21. 21. Edit Propagation User Edits
    22. 22. Edit Propagation User Edits
    23. 23. Edit Propagation User Edits
    24. 24. Edit Propagation User Edits
    25. 25. Edit Propagation User Edits
    26. 26. User Edits Edit Propagation
    27. 27. Edited Video Edit Propagation
    28. 28. Super-Resolution
    29. 29. Conclusion <ul><li>Spacetime fusion </li></ul>
    30. 30. Conclusion <ul><li>Spacetime fusion </li></ul><ul><ul><li>Combines spatial and temporal gradients from two different sources </li></ul></ul>
    31. 31. Conclusion <ul><li>Spacetime fusion </li></ul><ul><ul><li>Combines spatial and temporal gradients from two different sources </li></ul></ul><ul><ul><li>Requires motion vectors for temporal source </li></ul></ul><ul><ul><ul><li>stereo (static scenes) </li></ul></ul></ul><ul><ul><ul><li>flow (dynamic scenes) </li></ul></ul></ul>
    32. 32. Conclusion <ul><li>Spacetime fusion </li></ul><ul><ul><li>Combines spatial and temporal gradients from two different sources </li></ul></ul><ul><ul><li>Requires motion vectors for temporal source </li></ul></ul><ul><ul><ul><li>stereo (static scenes) </li></ul></ul></ul><ul><ul><ul><li>flow (dynamic scenes) </li></ul></ul></ul><ul><ul><li>Major applications </li></ul></ul><ul><ul><ul><li>Enforcing temporal coherence </li></ul></ul></ul><ul><ul><ul><li>Transferring lighting information </li></ul></ul></ul>

    ×