1. Comparative Analysis of Novel
Boundary Detection Methods
ANIL ULAŞ KOÇAK - OSMAN BUĞRA SARICA
Image Processing Course, 2016, Ankara, Turkey
1
2. Plan
What is Boundary Detection?
Related Works
Boundary Detection Methods
Sketch Tokens
Structured Random Forest
Oriented Edge Forest
Crisp Boundary Detection
Comparison
Results
2
3. Boundary detection
3
Reduce dimensionality of data
Preserve content information
Useful in applications such as:
4. Boundary detection
4
Boundary detection is usually formulated as a per-pixel
classification problem
How to extract discriminative boundary features?
How to learn a efficient boundary classifier?
12. Boundary detection methods
Learning based contour detection:
Sketch Tokens: A Learned Mid-Level Representation for
Contour and Object Detection
Structured Forests for Fast Edge Detection
Oriented Edge Forest for Boundary Detection
Measuring rarity based on pointwise mutual information:
Crisp Boundary Detection Using Pointwise Mutual Information
12
14. Sketch tokens
A novel approach to both learning and detecting local edge-
based mid-level features. Having 2 main parts;
◦ Defining Sketch Tokens Classes
◦ defining clasess by clustering of the patches sampled from the human generated hand drawn
images
◦ Detecting Sketch Tokens
◦ Training a random forest classifier with feature information from training images
14
15. Sketch tokens
Defining Sketch Tokens Classes
Goal is to define a set of token classes that represent the wide variety of local
edge structures that may exist in an image.
15
16. Sketch tokens
Defining Sketch Tokens Classes
◦ It is defined by clustering of the patches sampled from the human generated
hand drawn images (novel approach)
◦ Only patches that contain a labeled contour at the center pixel are used
(35x35)
16
17. Sketch tokens
Detecting Sketch Tokens
◦ Feature Extraction
◦ Integral Channel Features
◦ Self Similarity Features
◦ Classification
◦ Using Random Forest Classifier to predict sketch tokens
17
20. Sketch tokens
20
Learning
Random Forest Classifiers.
Advantages:
Fast at test time, especially for
a nonlinear classifier.
Dont have to explicitly compute
independent descriptors for
every patch. Just look up what
the decision tree wants to
know at each branch.
21. Sketch tokens
Detections of individual sketch tokens
21
Combining sketch token
detections:
Random forest classifier predicts the
probability that an image patch
belongs to each token class or the
negative set.
22. Crisp boundary detection
Main Idea: pixels belonging to the same object show higher statistical
dependencies than pixels belonging to the different object.
23. Crisp boundary detection
black-next-to-white occurs over and over again. This pattern shows up
in the image’s statistics as a suspicious coincidence — these colors must
be part of the same object!
23
24. Crisp boundary detection
Pointwise Mutual Information (PMI) is used to obtain
statistical association between two pixel and get affinity
measure, so can be predicted whether or not two pixel lie on
same object.
24
26. Crisp boundary detection
Accurate Result
◦ Thanks to good prediction of PMI distinguishing between boundary
and non boundary is provided.
Crisp Result
◦ Highly localized features (only color and color variance information is
used in 3x3 window)
26
27. Structured forests
27
Idea
It is an enhancement from Sketch token
Using the edge structure directly instead of using predefined label
It might allows it to learn more subtle variations in edge structure and leads to
a more accurate and efficient algorithm.
Structured learning with random forest
It did not cluster the edges before training the tree
If the splitting function could work with the edge structure directly,
then the random forest algorithm could work in original way
29. Difference 2
Difference 1
Structured forests
29
Structured random forest
Slightly modified from normal random forest
Clustering the incoming patches into 2 groups
based on the Z representation.
Now the patches have labels! Thus we could
still calculate the information gain to choose
the feature f and threshold t.
The representative edge in the leaf node is
chosen by:
The edge structure whose Z representation is the
medoid
30. Structured forests
30
Generate the edge map.
Each prediction is a voting. The pixel with higher votes has
higher confidence of being an edge
32. Oriented edge forest
32
Once a forest is trained to recognize oriented edge
patterns, it is applied over the input image in scanning-
window fashion. From a test patch x each tree produces a
distribution which are then combined either by
averaging (accuracy) or voting (speed, sparsity).
where 1 is the indicator function.
33. Oriented edge forest
33
Method provide a simple procedure for calibrating forest-
generated posterior probabilities.
34. Oriented edge forest
34
The forest produces calibrated distributions at every
spatial position. To derive edge strengths we composite
sharpened edge masks into the image, weighted by their
posterior probabilities.
36. Evaluation Metric
IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), 2015 5
Precision-Recall Curve
Precision =
𝑇𝑝
𝑇𝑝+𝐹𝑝
How many output results are relevant ?
Recall =
𝑇𝑝
𝑇𝑝+𝐹𝑛
How many relevant output results are selected ?
Time cost
Measuring time during selected scenario.
Complexity
Having complex or simple understanding
43. Conclusion
IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), 2015 24
Method Pros Cons
Sketch Tokens Having Inovative
approach
Relatively Slow and
Average Success
Crisp B. D. Great Success,
Simple approach
Slow
Structured Good success on
performance and
time cost
Complex
Oriented Having relatively
good time cost
Not successful by
year , Complex
Editor's Notes
Canny :
Muhtemelen computer visionda en çok kullanılan edge detector.
Filter image with x, y derivatives of Gaussian
Find magnitude and orientation of gradient
Non maximum suppression:
Thin multi-pixel wide «ridges» down to single width
Thresholding and linking(hysteresis):
Define two thresholds: low and high
Use the high threshold to start edge curves and the low threshold to continue them.
Goal: “use features extracted from such an image patch to estimate the posterior probability of a boundary passing through the center point”
Use cues such as intensity, brightness, color and texture to get a measure of boundary strength
How to combine cues? It’s a supervised learning problem.
Learn an optimal local boundary model from labeled images
Approach: look at each pixel for local discontinuities in several feature channels, over a range of orientations and scales
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
The image boundary connectivity cue has been proved very effective through several state-of-the-art methods
To compute saliency, several bottom-up cues have been used in previous literature:
One of them is the contrast/rarity cue, that assumes rare appearance tends to be more saliency
Another one, called Image Boundary Connectivity cue, has been used in many recent methods. It assumes that background elements are usually extends beyond the image boundary, and thus connected to the image boundary. In contrast, salient objects are often centered and have closed boundaries.
The image boundary connectivity cue has been proved very effective through several state-of-the-art methods
The image boundary connectivity cue has been proved very effective through several state-of-the-art methods
The image boundary connectivity cue has been proved very effective through several state-of-the-art methods
The image boundary connectivity cue has been proved very effective through several state-of-the-art methods
The image boundary connectivity cue has been proved very effective through several state-of-the-art methods
The image boundary connectivity cue has been proved very effective through several state-of-the-art methods
Given an input image:
For each channel in the Lab color space:
Apply Minimum Barrier Distance transform to compute MBD maps, which measure image boundary connectivity of each pixel
Average the MBD maps of the color channels
Apply postprocessing to improve the saliency map quality for object segmentation
Optionally, we can further enhance the saliency map by leveraging the Backgroundness cue at a moderately increased cost. Backgroundness assumes image boundary regions are mostly background.
Given an input image:
For each channel in the Lab color space:
Apply Minimum Barrier Distance transform to compute MBD maps, which measure image boundary connectivity of each pixel
Average the MBD maps of the color channels
Apply postprocessing to improve the saliency map quality for object segmentation
Optionally, we can further enhance the saliency map by leveraging the Backgroundness cue at a moderately increased cost. Backgroundness assumes image boundary regions are mostly background.
How to measure image boundary connectivity by distance transform:
Set image boundary pixels as seed set (show in red)
For each pixel (show in greed), find the shortest path (show in grey) to the seed set, according to the given path cost function
The cost of the shortest path is the distance between green and red.
How to measure image boundary connectivity by distance transform:
Set image boundary pixels as seed set (show in red)
For each pixel (show in greed), find the shortest path (show in grey) to the seed set, according to the given path cost function
The cost of the shortest path is the distance between green and red.
Most of the state-of-the-art methods requires super-pixel representation of the image. Computation of the super-pixel becomes the speed bottleneck.
Some sample saliency maps
The baseline GD, which uses Geodesic Distance, often produces fuzzy central area.
