Class-Specific, Top-Down Segmentation Eran Borenstein and Shimon Ullman Presenter : Rafi Zachut Instructor :  Lior Wolf
Goal <ul><li>To create figure-ground map using class-specific criteria </li></ul>Motivation <ul><li>Human vision </li></ul>
Bottom-up Segmentation <ul><li>Rely on image based criteria: </li></ul><ul><ul><li>grey level or texture uniformity  </li>...
Difficulty of class-specific segmentation <ul><li>Large variability of shapes within a given class </li></ul>Solution <ul>...
Example
Training – Searching the fragments  <ul><li>Generate large number of candidates from class images (C) </li></ul><ul><li>2....
Fragment information <ul><li>Grey level template (left) </li></ul><ul><li>Figure – ground label (right) </li></ul><ul><li>...
Segmentation by Optimal Cover  <ul><li>A cover fragments define a  figure–ground  map </li></ul><ul><li>An “optimal cover”...
Individual Match <ul><li>Similarity between a fragment and the region it covers </li></ul><ul><li>Combines region normaliz...
Individual Match – Cont’d
Consistency <ul><li>Cover quality from global view </li></ul><ul><li>Consistency between two overlapping fragments F i  an...
Fragment Reliability <ul><li>Reliable fragments guide the covering (similar to a puzzle) </li></ul>
A Cover Score <ul><li>A cover is an assignment of fragments to positions : </li></ul><ul><li>Its score is defined by : </l...
Finding the Optimal Score <ul><li>Finding the optimal cover is exhausting  </li></ul><ul><li>Instead we use greedy iterati...
The Cover Algorithm <ul><li>Pre-Processing </li></ul><ul><ul><li>For each image position find best fragment  (Position Sco...
Scaling <ul><li>No analytical support  </li></ul><ul><li>Pre-processing stage is applied on 5 different scales of the targ...
Results
Advantages <ul><li>Automatic training </li></ul><ul><li>Meaningful approximation for the figure-ground map of the image  <...
Combining Top-down and Bottom-up segmentation Eran Borenstein Eitan Sharon  Shimon Ullman
Bottom-up vs. Top-down Inaccurate boundaries Accurate boundaries Figure-ground approx. Multiple segments Use class informa...
Bottom-up in a nutshell <ul><li>Applies successive recursive image coarsening </li></ul><ul><li>Homogenous segments at one...
Bottom-up segmentation tree <ul><li>Nodes  – segments at different coarsening levels  </li></ul><ul><li>Arcs     – connect...
Bottom-up Saliency Measurement <ul><li>Ranks segments by their distinctiveness </li></ul><ul><li>Roughly: </li></ul>low sa...
Top-down & Bottom-up - Goal <ul><li>To provide accurate figure-ground map </li></ul>Top - down Top – down & Bottom-up
The Approach  <ul><li>Work is constrained to using Bottom-up segments. Actually the labeling (figure/ground) of the tree l...
Top-down & Bottom-up conflict <ul><li>Top-down wants the final map to be as close as possible to its map  </li></ul><ul><l...
The Top-down cost <ul><li>If the final map (defined by the leaves labeling) is  C  and the Top-down map is  T ,  then the ...
The Total cost <ul><li>Is simply a weighted sum of both costs, i.e.  </li></ul><ul><li>║ C - T║ 2  +  λ  ∑ b i   </li></ul...
Minimizing the cost function <ul><li>If  N  is the number of segments in the tree, there are  2 N  labeling options </li><...
Confidence map <ul><li>After solving each segment S holds two values: </li></ul><ul><ul><li>m s (+1) -  minimum cost when ...
Results
Conclusion <ul><li>General approach to use Top-down info to : </li></ul><ul><ul><li>Group together segments belonging to t...
<ul><li>Thank You </li></ul>
Upcoming SlideShare
Loading in...5
×

Rafi Zachut's slides on class specific segmentation

790

Published on

Class-Specific, Top-Down Segmentation. Eran Borenstein and Shimon Ullman.
Combining Top-down and Bottom-up segmentation
Eran Borenstein
Eitan Sharon
Shimon Ullman

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
790
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Rafi Zachut's slides on class specific segmentation

  1. 1. Class-Specific, Top-Down Segmentation Eran Borenstein and Shimon Ullman Presenter : Rafi Zachut Instructor : Lior Wolf
  2. 2. Goal <ul><li>To create figure-ground map using class-specific criteria </li></ul>Motivation <ul><li>Human vision </li></ul>
  3. 3. Bottom-up Segmentation <ul><li>Rely on image based criteria: </li></ul><ul><ul><li>grey level or texture uniformity </li></ul></ul><ul><ul><li>smoothness and continuity of bounding contours </li></ul></ul>
  4. 4. Difficulty of class-specific segmentation <ul><li>Large variability of shapes within a given class </li></ul>Solution <ul><li>Use fragment-based representation (extracted from training examples) </li></ul><ul><li>Cover novel object with fragments (like puzzle) to define the figure-ground map </li></ul>
  5. 5. Example
  6. 6. Training – Searching the fragments <ul><li>Generate large number of candidates from class images (C) </li></ul><ul><li>2. For every fragment F i : </li></ul><ul><ul><li>Find max correlation S i for each image in C and NC (non class images) </li></ul></ul><ul><ul><li>Set detection threshold θ i such that : </li></ul></ul><ul><li>p (S i > θ i | NC) ≤ α </li></ul><ul><li>3. Select K fragments with best p (S i > θ i | C) </li></ul>
  7. 7. Fragment information <ul><li>Grey level template (left) </li></ul><ul><li>Figure – ground label (right) </li></ul><ul><li>Reliability value : p (S i > θ i | C) </li></ul>
  8. 8. Segmentation by Optimal Cover <ul><li>A cover fragments define a figure–ground map </li></ul><ul><li>An “optimal cover” is needed </li></ul><ul><li>Quality of a cover is measured by: </li></ul><ul><ul><li>Individual Match </li></ul></ul><ul><ul><li>Consistency </li></ul></ul><ul><ul><li>Fragment Reliability </li></ul></ul>
  9. 9. Individual Match <ul><li>Similarity between a fragment and the region it covers </li></ul><ul><li>Combines region normalized correlation with edge detection : </li></ul>
  10. 10. Individual Match – Cont’d
  11. 11. Consistency <ul><li>Cover quality from global view </li></ul><ul><li>Consistency between two overlapping fragments F i and F j is defined as : </li></ul>
  12. 12. Fragment Reliability <ul><li>Reliable fragments guide the covering (similar to a puzzle) </li></ul>
  13. 13. A Cover Score <ul><li>A cover is an assignment of fragments to positions : </li></ul><ul><li>Its score is defined by : </li></ul>
  14. 14. Finding the Optimal Score <ul><li>Finding the optimal cover is exhausting </li></ul><ul><li>Instead we use greedy iterative algorithm that converges to a local maximum </li></ul><ul><li>Typically 2-3 iterations </li></ul><ul><li>Complexity-linear in the image size and the number of fragments </li></ul>
  15. 15. The Cover Algorithm <ul><li>Pre-Processing </li></ul><ul><ul><li>For each image position find best fragment (Position Score = Individual Match * Reliability) </li></ul></ul><ul><ul><li>Select the sub-window with maximum score of M positions </li></ul></ul><ul><li>Iteration </li></ul><ul><ul><li>Choose the best M unused fragments </li></ul></ul><ul><ul><li>Add the subset that maximizes the score </li></ul></ul><ul><ul><li>Remove all fragments that reduce the score </li></ul></ul>
  16. 16. Scaling <ul><li>No analytical support </li></ul><ul><li>Pre-processing stage is applied on 5 different scales of the target image and the algorithm continues with the best window in the best scale </li></ul>
  17. 17. Results
  18. 18. Advantages <ul><li>Automatic training </li></ul><ul><li>Meaningful approximation for the figure-ground map of the image </li></ul><ul><li>Can be extended to support multiple classes </li></ul>Disadvantages <ul><li>Covering highly variable parts is difficult </li></ul><ul><li>Inaccurate delineation of object boundaries </li></ul>
  19. 19. Combining Top-down and Bottom-up segmentation Eran Borenstein Eitan Sharon Shimon Ullman
  20. 20. Bottom-up vs. Top-down Inaccurate boundaries Accurate boundaries Figure-ground approx. Multiple segments Use class information Rely on image criteria
  21. 21. Bottom-up in a nutshell <ul><li>Applies successive recursive image coarsening </li></ul><ul><li>Homogenous segments at one level are used to form larger segments at the next level </li></ul><ul><li>i.e. the image is segmented into fewer and fewer segments </li></ul><ul><li>Complexity - linear in image pixels </li></ul>
  22. 22. Bottom-up segmentation tree <ul><li>Nodes – segments at different coarsening levels </li></ul><ul><li>Arcs – connect between a segment and its sub- segments at a finer level </li></ul>
  23. 23. Bottom-up Saliency Measurement <ul><li>Ranks segments by their distinctiveness </li></ul><ul><li>Roughly: </li></ul>low saliency high saliency Internal homogeneity Dissimilarity with the surrounding saliency + =
  24. 24. Top-down & Bottom-up - Goal <ul><li>To provide accurate figure-ground map </li></ul>Top - down Top – down & Bottom-up
  25. 25. The Approach <ul><li>Work is constrained to using Bottom-up segments. Actually the labeling (figure/ground) of the tree leaves defines the final map </li></ul><ul><li>Use Top-down approximation to overcome Bottom-up problems: </li></ul><ul><ul><li>Group together segments belonging to the object despite image-based dissimilarity </li></ul></ul><ul><ul><li>Break apart homogenous segments contain both figure and ground regions </li></ul></ul>
  26. 26. Top-down & Bottom-up conflict <ul><li>Top-down wants the final map to be as close as possible to its map </li></ul><ul><li>Bottom-up wants to keep salient segments complete </li></ul><ul><li>When the Top-down map crosses a salient segment a decision is need to be taken: </li></ul><ul><li>breaking the salient segment versus distancing from the Top-down map </li></ul>
  27. 27. The Top-down cost <ul><li>If the final map (defined by the leaves labeling) is C and the Top-down map is T , then the Top-down cost is : ║C - T║ 2 </li></ul>Bottom-up cost <ul><li>Each segment S i labeled differently from its parent will pay b i = │ S i │ (1- h i ) where h i is its saliency (between 0 to 1) </li></ul><ul><li>Low-salience segments pay more (they break salient segments) </li></ul>
  28. 28. The Total cost <ul><li>Is simply a weighted sum of both costs, i.e. </li></ul><ul><li>║ C - T║ 2 + λ ∑ b i </li></ul><ul><li>Top-down requirement is constrained by the bottom-up tree </li></ul><ul><li>The variables are the segments labels </li></ul>
  29. 29. Minimizing the cost function <ul><li>If N is the number of segments in the tree, there are 2 N labeling options </li></ul><ul><li>Luckily the cost function can be solved by the product-sum algorithm in O(N) </li></ul><ul><li>See attached example </li></ul>
  30. 30. Confidence map <ul><li>After solving each segment S holds two values: </li></ul><ul><ul><li>m s (+1) - minimum cost when S is figure </li></ul></ul><ul><ul><li>m s (-1) - minimum cost when S is ground </li></ul></ul><ul><li>The confidence in segment S labeling is: </li></ul><ul><ul><ul><li>│ m s (+1) - m s (-1) │ / │ S │ </li></ul></ul></ul><ul><li>If the minimum cost is not sensitive to the labeling of S, the confidence is low </li></ul><ul><li>A confidence map can be constructed from the confidence of the leaves </li></ul>
  31. 31. Results
  32. 32. Conclusion <ul><li>General approach to use Top-down info to : </li></ul><ul><ul><li>Group together segments belonging to the object despite image-based dissimilarity </li></ul></ul><ul><ul><li>Break apart homogenous segments contain both figure and ground regions </li></ul></ul><ul><li>Efficient algorithm – linear in the tree nodes </li></ul><ul><li>Reliable confidence map with no extra computation </li></ul>
  33. 33. <ul><li>Thank You </li></ul>

×