![Imperceptible Motions and Changes
[Liu et al. 2005] [Wu et al. 2012]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
Similar to Motion Amplification PhaseBasedSIGGRAPH2013pres.pptx
Similar to Motion Amplification PhaseBasedSIGGRAPH2013pres.pptx (20)
Recently uploaded
Recently uploaded (20)
Motion Amplification PhaseBasedSIGGRAPH2013pres.pptx
- 1. Phase-Based Video Motion Processing Neal Wadhwa Michael Rubinstein Fredo Durand William T. Freeman MIT Computer Science and Artificial Intelligence Lab (CSAIL)
- 2. Imperceptible Motions and Changes [Liu et al. 2005] [Wu et al. 2012]
- 3. MAGNIFIED Imperceptible Motions and Changes [Wu et al. 2012] [Liu et al. 2005]
- 4. Motion Magnification [Liu et al. SIGGRAPH 2005] [Liu et al. 2005] Source Motion-magnified
- 5. Lagrangian Motion Magnification • Each particle (pixel) is tracked • Motion vectors are amplified directly
- 6. Lagrangian Motion Magnification Tracking Motion layer segmentation Inpainting
- 7. Lagrangian Motion Magnification Tracking Motion layer segmentation Inpainting
- 8. Eulerian Video Magnification [Wu et al. SIGGRAPH’12] • Examine the varying values at a fixed voxel grid (video) • We treat each pixel as a time series and apply signal processing to it
- 9. Eulerian Video Magnification [Wu et al. SIGGRAPH’12]
- 10. EVM in the Wild: YouTube “Tomez85” Institute for Biomedical Engineering Dresden Germany “SuperCreaturefan” Source Motion-magnified Source Motion-magnified
- 11. EVM in the Wild: Web App
- 12. Limitations Source (300 fps) Source Phase-based (this paper) Intensity clipping Amplified noise Source [Wu et al. 2012] [Wu et al. 2012]
- 13. Talk Overview • Eulerian Video Magnification [Wu et al. SIGGRAPH’12] – Hao-yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, William T. Freeman • Phase-Based Video Motion Processing [this paper] – Neal Wadhwa, Michael Rubinstein, Frédo Durand, William T. Freeman • Results, new applications, controlled sequences
- 14. Talk Overview • Eulerian Video Magnification [Wu et al. SIGGRAPH’12] – Hao-yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, William T. Freeman • Phase-Based Video Motion Processing [this paper] – Neal Wadhwa, Michael Rubinstein, Frédo Durand, William T. Freeman • Results, new applications, controlled sequences
- 15. Eulerian Video Magnification Pipeline [Wu et al. 2012] Spatial Decomposition Spatial Decomposition Temporal filtering Spatial Decomposition Temporal filtering α1 α2 αn-1 αn Σ Σ Σ Σ Spatial Decomposition Temporal filtering α1 α2 αn-1 Reconstruction αn Σ Σ Σ Σ
- 16. Why It Amplifies Motion
- 17. Differential Brightness Constancy 0 𝜕𝑥 𝜕𝑡 𝜕𝑓 𝜕𝑡 𝜕𝑓 𝜕𝑥 Signal at time 𝑡 Signal at time 𝑡 + Δ𝑡 Intensity Space (𝑥)
- 18. Differential Brightness Constancy 0 𝜕𝑓 𝜕𝑥 Signal at time 𝑡 Signal at time 𝑡 + Δ𝑡 Space (𝑥) Intensity 𝛼 𝜕𝑓 𝜕𝑡 Implied Translation 𝜶𝜹 𝜶 𝝏𝒇 𝝏𝒕 ≈ 𝜶𝜹 𝝏𝒇 𝝏𝒙 1st-order linear approximation:
- 19. Relating Temporal and Spatial Changes Courtesy of Lili Sun Intensity 0 255 Space (𝑥) Signal at time 𝑡 Signal at time 𝑡 + Δ𝑡 Motion-magnified
- 20. Limitations of Linear Motion Processing Intensity 0 255 Overshoot Space (𝑥) Undershoot Signal at time 𝑡 Signal at time 𝑡 + Δ𝑡 Motion-magnified • Breaks at high spatial frequencies and large motions
- 21. Limitations of Linear Motion Processing Intensity Space (𝑥) • Noise amplified with signal Signal at time 𝑡 + Δ𝑡 Motion-magnified 𝜕𝑥 𝜕𝑡 𝜕𝑓 𝜕𝑡 𝜕𝑓 𝜕𝑥
- 22. Limitations of Linear Motion Processing Source Linear SIGGRAPH’12 Phase-based SIGGRAPH’13 Overshoot Overshoot
- 23. Linear vs. Phase-Based Motion Processing • Linear Motion Processing – Assumes images are locally linear – Translate by changing intensities • NEW Phase-Based Motion Processing – Represents images as collection of local sinusoids – Translate by shifting phase 𝜕𝑥 𝜕𝑡 𝜕𝑓 𝜕𝑡 𝜕𝑓 𝜕𝑥
- 24. Linear vs. Phase-Based Motion Processing Source Linear SIGGRAPH’12 Phase-based SIGGRAPH’13
- 25. Talk Overview • Eulerian Video Magnification [Wu et al. SIGGRAPH’12] – Hao-yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, William T. Freeman • Phase-Based Video Motion Processing [this paper] – Neal Wadhwa, Michael Rubinstein, Frédo Durand, William T. Freeman • Results, new applications, controlled sequences
- 26. • For illustration, let’s look at a 1D image profile Fourier Decomposition FFT Space (x) Intensity 𝐴1 × Space (x) Intensity + ⋯ +𝐴2 × Intensity Space (x) 𝑓 𝑥 𝜔=−∞ ∞ 𝐴𝜔𝑒𝑖𝜔𝑥 FFT
- 27. Amplitude of Basis Function 𝐴1 × +𝐴2 × + ⋯ 𝑆𝜔 Amplitude (𝑆1) Amplitude (𝑆2) Space (x) Space (x) Intensity Intensity Intensity Space (x) FFT 𝑓 𝑥 𝜔=−∞ ∞ 𝐴𝜔𝑒𝑖𝜔𝑥 FFT
- 28. • Phase controls location of sinusoid 𝑓 𝑥 𝑓 𝑥 − 𝛿 Fourier Shift Theorem 𝐴1 × +𝐴2 × + ⋯ 𝜔=−∞ ∞ 𝐴𝜔𝑒𝑖𝜔𝑥 FFT 𝑒−𝑖𝜔𝛿 Phase (𝑒−𝑖𝛿 ) Phase(𝑒−𝑖2𝛿 ) Space (x) Space (x) Intensity Intensity Phase Shift ↔ Translation Intensity Space (x) FFT
- 29. Local Motions • Fourier shift theorem only lets us handle global motion • But, videos have many local motions • Need a localized Fourier Series for local motion Mast Hook Building
- 30. Complex Steerable Pyramid [Simoncelli et al. 1992] Filter Bank Orientation 2 Real Imag Scale 1 Orientation 1 Scale 2 Orientation 1 Orientation 2 Frequency (𝜔 𝑦 ) Frequency (𝜔𝑥) Idealized Transfer Functions Scale Orientation FFT
- 31. Complex Steerable Pyramid Basis Functions × Window Intensity Space (x) Complex Sinusoid (Global) Intensity Space (x) Space (x) Intensity
- 32. Single Sub-Band (Scale) • In single scale, image is coefficients times translated copies of basis functions 𝐴1 × +𝐴2 × + ⋯ Single Sub-band of Image Profile =
- 33. Single Sub-Band (Scale) • In single scale, image is coefficients times translated copies of basis functions 𝐴1 × +𝐴2 × + ⋯ Single Sub-band of Image Profile =
- 34. Local Amplitude • Local amplitude controls strength of basis function 𝐴1 × +𝐴2 × + ⋯ Single Sub-band of Image Profile = Local Amplitude
- 35. Local Phase • Local phase controls location of sinusoid under window, approximates local translation +𝐴2 × + ⋯ Single Sub-band of Image Profile = Local Phase Local Phase Shift ↔ Local Translation Local Motions 𝐴1 ×
- 36. • Phase-based motion synthesis • Phase-based optical flow Phase and Motion [Fleet and Jepson 1990] [Gautama and Van Hulle 2002] Motion without Movement [Freeman et al. 1991]
- 37. Time (t) Phase over Time Radians Time (t) Time (t) Radians Phase over Time … Wavelets Input Space(x) Intensity Intensity Space(x) a Intensity Space(x)
- 38. Phase over Time Time (t) Phase over Time Input Motion-magnified Space(x) Radians Time (t) Time (t) Radians Space(x) Intensity Intensity … Space(x) Intensity Intensity Space(x) Wavelets
- 39. 2D Complex Steerable Pyramid Filter Bank Orientation 2 Real Imag Scale 1 Orientation 1 Scale 2 Orientation 1 Orientation 2 Frequency (𝜔 𝑦 ) Frequency (𝜔𝑥) Idealized Transfer Functions Scale Orientation FFT Hipass Residual Lopass Residual
- 40. Complex Steerable Pyramid Decomposition Amplitude Sub-bands Filter Bank Orientation 2 Real Imag Scale 1 Orientation 1 Scale 2 Orientation 1 Orientation 2 Phase Amplitude Phase 𝐴 𝑒𝑖𝜙 ×
- 41. Phase over Time Amplitude Sub-bands Filter Bank Orientation 2 Real Imag Scale 1 Orientation 1 Scale 2 Orientation 1 Orientation 2 Phase Phase Time (s) Phase over time Phase Bandpassed Phase over time Time (s) Temporal Filtering
- 42. New Phase-Based Pipeline Amplitude Sub-bands Filter Bank Complex steerable pyramid [Simoncelli et al. 1992] Bandpassed Phase Orientation 2 Real Imag Scale 1 Orientation 1 Scale 2 Orientation 1 Orientation 2 Phase Reconstruction 𝛼 ⊗ 𝛼 ⊗ 𝛼 ⊗ 𝛼 ⊗ Temporal Filtering Temporal filtering on phases
- 43. Linear Pipeline (Wu et al. 2012) Laplacian pyramid [Burt and Adelson 1983] Temporal filtering on intensities
- 44. New Phase-Based Pipeline Amplitude Sub-bands Filter Bank Complex steerable pyramid [Simoncelli et al. 1992] Bandpassed Phase Orientation 2 Real Imag Scale 1 Orientation 1 Scale 2 Orientation 1 Orientation 2 Phase Reconstruction 𝛼 ⊗ 𝛼 ⊗ 𝛼 ⊗ 𝛼 ⊗ Temporal Filtering Temporal filtering on phases
- 45. Improvement #1: Less Noise Noise amplified Noise translated Source (IID Noise, std=0.1) Linear [Wu et al. 2012] (x50) Phase-based (x50)
- 46. Improvement #2: More Amplification Amplification factor Motion in the sequence Range of linear method: Range of phase-based method: 4 times the amplification!
- 47. Attenuated • Local phase can move image features, but only within the filter window Limits of Phase Based Magnification Amplification factor
- 48. See Paper For… • The bound on amplification • Sub-octave bandwidth pyramid 𝛼𝛿 < 𝜆 2 Amplification Initial motion Spatial wavelength Compact Overcomplete
- 49. Comparison with [Wu et al. 2012] Wu et al. 2012 Phase-Based (this paper)
- 50. Vibration due to Camera’s Mirror Source (300 FPS) Wu et al. 2012 Phase-based (this paper)
- 51. Comparison with [Wu et al. 2012] and Video Denoising Wu et al. with VBM3D Phase-based (this paper) Wu et al. Wu et al. + Liu and Freeman 2010
- 52. Talk Overview • Eulerian Video Magnification [Wu et al. SIGGRAPH’12] – Hao-yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, William T. Freeman • Phase-Based Video Motion Processing [this paper] – Neal Wadhwa, Michael Rubinstein, Frédo Durand, William T. Freeman • Results, new applications, controlled sequences
- 53. Eye Movements Source (500FPS) Motion magnified x150 (30-50 Hz)
- 54. Expressions Low frequency motions Mid-range frequency motions Source
- 55. Ground Truth Validation • Induce motion (with hammer) • Record with accelerometer Accelerometer Hammer Hit
- 57. Qualitative Comparison Input (motion of 0.1 px) “Ground truth” (motion of 5 px) Motion-magnified (x50) time space
- 58. Vibration Modes Slide removed due to licensing restrictions.
- 59. Motion Attenuation Source Turbulence Removed Sequence courtesy Vimeo user Vincent Laforet
- 64. Neck Skin Vibrations Frequency (Hz) 0 500 1000 Power Source (2 KHz)
- 65. Source (2 KHz) Frequency (Hz) 0 500 1000 Power Source (2 KHz) 100 Hz Amplified x100 Fundamental frequency: ~100Hz
- 66. Source (2 KHz) Amplified (x100)
- 67. Conclusions • New representation for analyzing and editing small motions • Much better than linear EVM [Wu et al. 2012] – Less noise – More amplification • Still “Eulerian” (no optical flow), but more explicit representation of motion – New capabilities (e.g. attenuating distracting motions) Linear SIGGRAPH’12 Phase-based SIGGRAPH’13 Phase over time
- 68. Phase-Based Motion Processing: Code and Web App • Code available soon: http://people.csail.mit.edu/nwadhwa/phase-video/ http://videoscope.qrclab.com/
- 69. Thank You! Input Input Input Input Magnified Magnified Magnified Magnified
- 70. Non-periodic Motions and Large Motions Source (300 FPS) Motion Magnification x50 Motion Magnification x50 Large Motions Unmagnified Non-periodic motion
- 71. Amplify Color without Motion Wu et al. 2012 Amplify Color without Motion
- 72. Denoising as a pre-process of [Wu et al. 2012] Linear [ Wu et al. 2012] NLM [ Buades et al. 2008] + Linear Liu and Freeman 2010 + Linear VBM3D + Linear Phase-based (this paper)
- 73. Comparison of Linear and Phase-Based Methods
- 74. Sub-octave Bandwidth Pyramids • Wider spatial filters yield better results in the crane sequence • But wider spatial filters means more levels to process resulting in higher cost • And multiple motions might be within a single basis function’s window. Octave Half Octave Quarter Octave Running time: 17s on 10s clip (280x280) Running time: 84s on 10s clip (280x280) Running time: 170s on 10s clip (280x280)