Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

mihara_iccp16_presentation

1,311 views

Published on

This is a slide for IEEE International Conference on Computational Photography (ICCP) 2016 in Northwestern University.
See for details: http://omilab.naist.jp/project/LFseg/

Published in: Science
  • Be the first to comment

mihara_iccp16_presentation

  1. 1. 4D Light Field Segmentation with Spatial and Angular Consistencies 1 Nara Institute of Science and Technology (NAIST), Japan. 2 Osaka University, Japan. 3 Kyushu University, Japan. http://omilab.naist.jp/project/LFseg Yasuhiro Mukaigawa1 Hajime Mihara1 Takuya Funatomi1 Kenichiro Tanaka2,1 Hiroyuki Kubo1 Hajime Nagahara3
  2. 2. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 2 Image editing is popular Take the scene with digital camera We can edit with great editing software. Photo by Eli Duke, https://flic.kr/p/fvn5qG editing with GIMP. Image data
  3. 3. “4D Light Field Segmentation with Spatial and Angular Consistencies” Light fields provide a very rich representation of a scene. 2016/5/23 3 Light field imaging is cool Refocusing Free view point image http://lightfield-forum.com/en/http://lightfield-forum.com/en/
  4. 4. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 4 Is light field editing popular? Can we edit light field? Take the scene with light field camera Light field data ?
  5. 5. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 5 Efficient light field editing with our method
  6. 6. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 6 Efficient light field editing with our method
  7. 7. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 7 Efficient light field editing with our method
  8. 8. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 8 Efficient light field editing with our method
  9. 9. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 9 Efficient light field editing with our method
  10. 10. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 10 Efficient light field editing with our method Input light field Edited light field
  11. 11. “4D Light Field Segmentation with Spatial and Angular Consistencies” “Editing light fields are challenging task.” [Jarabo+, ToG ’14]  Three difficulties: I. There are no interfaces for 4D editing. II. Software must use depth to minimize redundancy for user. III. The local edit on a light field needs to preserve the coherency.  Our solution I. Ask user to input seeds on !!only!! central 2D image II. Propagate seeds to other images based on color and estimated depth 2016/5/23 11 Problem statement
  12. 12. “4D Light Field Segmentation with Spatial and Angular Consistencies” “Editing light fields are challenging task.” [Jarabo+, ToG ’14]  Three difficulties: I. There are no interfaces for 4D editing. II. Software must use depth to minimize redundancy for user. III. The local edit on a light field needs to preserve the coherency.  Our solution I. Ask user to input seeds on !!only!! central 2D image II. Propagate seeds to other images based on color and estimated depth 2016/5/23 12 Problem statement
  13. 13. “4D Light Field Segmentation with Spatial and Angular Consistencies” Solution:  Require the user input for only the center image of 4D light field. 2016/5/23 13 Interface for light field editing Input: light field and Seeds for one image Output: labeled light field
  14. 14. “4D Light Field Segmentation with Spatial and Angular Consistencies” “Editing light fields are challenging task.” [Jarabo+, ToG ’14]  Three difficulties: I. There are no interfaces for 4D editing. II. Software must use depth to minimize redundancy for user. III. The local edit on a light field needs to preserve the coherency.  Our solution I. Ask user to input seeds on central 2D image II. Propagate seeds to other images based on color and estimated depth 2016/5/23 14 Problem statement
  15. 15. “4D Light Field Segmentation with Spatial and Angular Consistencies” In 2D: Color feature (histogram) In 4D: Color and depth feature can be used.  Objectness : likelihoods that the ray will have the label which is defined by color and depth for robust segmentation 15 Effective use of depth: Objectness Color Depth Positive samples Negative samples “foreground” depth estimation • [Wanner+, ‘12] • [Chen+, ‘14] • [Lin+, ‘15] • [Wang+, ‘15] [Wanner+, ’12] Feature Vector [Color, depth] Given input for one image Training SVM High Low Objectness “foreground” “background”
  16. 16. “4D Light Field Segmentation with Spatial and Angular Consistencies” Objectness for multi label segmentation  Use one-vs-rest SVMs 2016/5/23 16 Effective use of depth: Objectness “Leaves 1”“butterfly” “Leaves 2” “Floor” High Low “butterfly” “Floor” “Leaves 1” “Leaves 2” Objectnesses Label
  17. 17. “4D Light Field Segmentation with Spatial and Angular Consistencies” “Editing light fields are challenging task.” [Jarabo+, ToG ’14]  Three difficulties: I. There are no interfaces for 4D editing. II. Interfaces must use depth to minimize redundancy for user. III. The local edit on a light field needs to preserve the coherency.  Our solution I. Ask user to input seeds on central 2D image II. Propagate seeds to other images based on color and estimated depth 2016/5/23 17 Problem statement
  18. 18. “4D Light Field Segmentation with Spatial and Angular Consistencies” Smoothness constraints between Spatial neighbor and Angular neighbor 2016/5/23 18 Smoothness constraints for 4D segmentation
  19. 19. “4D Light Field Segmentation with Spatial and Angular Consistencies”  Our solution I. Ask user to input seeds on central 2D image II. Propagate seeds to other images based on color and estimated depth III. Introduce smoothness constraints for spatial and angular neighbors 2016/5/23 19 Solve  To combine these solutions:  State light field segmentation as energy minimization problem
  20. 20. “4D Light Field Segmentation with Spatial and Angular Consistencies” Energy function 2016/5/23 20 Energy minimization using graph cut Data term Rp(lp=“class i”) Determined by objectness Smoothness term Bp,q Assumption that neighboring rays have same label. Neighboring raysA ray Assigned labels Min. Data term Smoothness term Via graph cut
  21. 21. “4D Light Field Segmentation with Spatial and Angular Consistencies”  For four datasets by Wanner et al.  Synthesized light fields with Blender  Datasets contain brush strokes and ground truth of segmentation  Reference: [Wanner+, CVPR ‘13]  Able to segment 2D image using 4D light field.  Based on Random forest for utilizing color and depth.  Give smoothness by total variation (TV). 2016/5/23 21 Experiments *[Wanner+, ``Datasets and Benchmarks for Densely Sampled 4D Light Fields’’, VMV ’13] butterfly Buddha StillLife Horses
  22. 22. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 22 Result: “Buddha” data set OursGround Truth Wanner+ Input Output
  23. 23. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 23 Result: “Horses” data set OursGround Truth Wanner+ Input Output
  24. 24. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 24 Segmentation accuracy Entire light field Ours Papillon 98.3 Buddha 97.6 StillLife 96.6 Horses 95.7
  25. 25. “4D Light Field Segmentation with Spatial and Angular Consistencies” Central image Wanner+ Ours 97.5 98.3 96.4 97.7 96.5 96.4 95.1 95.9 2016/5/23 25 Segmentation accuracy Entire light field Ours Papillon 98.3 Buddha 97.6 StillLife 96.6 Horses 95.7
  26. 26. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 26 Segmentation accuracy Entire light field Ours Papillon 98.3 Buddha 97.6 StillLife 96.6 Horses 95.7 From the result: • Our method can segment the entire light field with much high accuracy. Central image Wanner+ Ours 97.5 98.3 96.4 97.7 96.5 96.4 95.1 95.9
  27. 27. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 27 Application: Efficient light field editing
  28. 28. “4D Light Field Segmentation with Spatial and Angular Consistencies” Input light field 2016/5/23 28 Efficient light field editing with our method
  29. 29. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 29 Efficient light field editing with our method Seeds
  30. 30. “4D Light Field Segmentation with Spatial and Angular Consistencies” Extract rays 2016/5/23 30 Efficient light field editing with our method
  31. 31. “4D Light Field Segmentation with Spatial and Angular Consistencies” Extract rays 2016/5/23 31 Efficient light field editing with our method
  32. 32. “4D Light Field Segmentation with Spatial and Angular Consistencies” Editing result 2016/5/23 32 Efficient light field editing with our method Input light field Edited light field
  33. 33. “4D Light Field Segmentation with Spatial and Angular Consistencies” Input light field 2016/5/23 33 Application: efficient light field editing
  34. 34. “4D Light Field Segmentation with Spatial and Angular Consistencies” Seeds 2016/5/23 34 Application: efficient light field editing
  35. 35. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 35 Application: efficient light field editing Extracted rays
  36. 36. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 36 Application: efficient light field editing Extracted rays
  37. 37. “4D Light Field Segmentation with Spatial and Angular Consistencies” Editing result 2016/5/23 37 Application: efficient light field editing Input light field Edited light field
  38. 38. “4D Light Field Segmentation with Spatial and Angular Consistencies”  We have developed supervised 4D light field segmentation  We solve three difficulties in light field editing: I. Ask user to input seeds on central 2D image II. Propagate seeds to other images based on color and estimated depth III. Introduce smoothness constraints for spatial and angular neighbors  Formulate problem as energy minimization  Show that our method can segment the entire light field with much high accuracy  Suggest that our method helps efficient light field editing2016/5/23 38 Summary
  39. 39. “4D Light Field Segmentation with Spatial and Angular Consistencies” Project Page: http://omilab.naist.jp/project/LFseg/ Points: 2016/5/23 39 Thank you for your attention! Learning based objectness from color and depth Two types of neighboring rays Segmented light field
  40. 40. “4D Light Field Segmentation with Spatial and Angular Consistencies”2016/5/23 40 EOF

×