SlideShare a Scribd company logo
AUTOMATIC DETECTION OF
COMPOUND STRUCTURES
FROM MULTIPLE HIERARCHICAL
SEGMENTATIONS
H¨useyin G¨okhan Akc¸ay
Department of Computer Engineering, Bilkent University, Bilkent, 06800, Ankara
akcay@cs.bilkent.edu.tr
21 Sept. 2016
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 1 / 78
Motivation
Large scale global content about the Earth.
Small local details (upto 30 cm resolution).
1.6 terabytes of data by the ESA’s multispectral
high-resolution imaging satellite.
The WorldView-2 satellite collects 975,000 square
kilometers of imagery per day.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 2 / 78
Motivation
A challenging problem in remote sensing image mining is
the detection of heterogeneous compound structures such
as different types of residential, industrial, and agricultural
areas.
Compound structures are comprised of spatial
arrangements of simple primitive objects such as buildings,
trees and road segments.
Detection of compound structures is a challenging problem
because
They contain thousands of primitive objects.
They mostly do not have distinctive features.
Primitives can arrange in many different combinations in the
overhead view.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 3 / 78
Motivation
Figure: 75 × 75m2 compound structures in WorldView-2 images.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 4 / 78
Literature Review
Primitive object detection.
Residential-factory buildings, local roads, vehicles, airplanes
and boats.
Window-based approaches.
Bag-of-words representation.
Enforces artificial boundaries on the image.
Assumes the whole window corresponds to a compound
structure.
Segmentation-based approaches.
The grouping criteria do not involve spatial arrangements.
Graph-based approaches.
Specific arrangements such as alignment and parallelism.
Structural graph matching.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 5 / 78
Problem Definition
We propose a generic method for the modeling and
detection of compound structures.
Target structures can involve arrangements of an unknown
number of different types of primitive objects.
The detection task is formulated as the selection of multiple
coherent subsets of candidate regions obtained from
multiple hierarchical segmentations.
To avoid over- or under-segmentation of candidate regions,
we search for the most meaningful regions at different
scales.
We propose a constrained region selection framework
which allows to specify global constraints on the selected
regions.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 6 / 78
Overview
Learning
Inference
Example
structure
Feature
extraction
Spatial
arrangement
model
Maximum
likelihood
estimation
Probabilistic
region
process
Selected
regions
Region
selection
Candidate
regions
Hierarchical
region
extraction
Image
Gibbs
sampling
S-W sampling/
QP
V
H(V)
p(V|β)
I
1
2
5 6
9 10
3 4
7
11
8
12 13 14
G = (V, E)
1
2
5 6
9 10
3 4
7
11
8
12 13 14
V∗
Figure: Object/process diagram of the proposed approach.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 7 / 78
Compound Structure Model
Primitive Representation
Compound structures are composed of spatial
arrangements of multiple, relatively homogeneous, and
compact primitive objects.
We assume that a compound structure V consists of R
layers of primitive object maps, V = r=1,...,R Vr
.
Figure: Primitive object layers.
Each primitive object vi is represented by an ellipse
vi = (li, si, θi).
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 8 / 78
Compound Structure Model
Spatial Arrangement Model
For a given compound structure consisting of N primitive
objects, we construct a neighborhood graph G = (V, E).
V = {v1, . . . , vN} correspond to the individual primitive
objects,
E = r1,r2=1,...,R Er1r2 where Er1r2 denotes the edges
between the vertices at layers Vr1 and Vr2 .
Figure: Neighborhood graph construction for multiple primitive layers.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 9 / 78
Compound Structure Model
Spatial Arrangement Model
For each (vi, vj) ∈ E, we compute the following five features:
Distance between the
closest pixels, φ1(vi, vj),
Relative orientation,
φ2(vi, vj),
φ2
φ3
φ1 φ4
Angle between the line joining the centroids of the two
objects and the major axis of vi as the reference object,
φ3(vi, vj),
Distance between the closest antipodal pixels that lie on the
major axes, φ4(vi, vj),
Relative size, φ5(vi, vj).
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 10 / 78
Compound Structure Model
Spatial Arrangement Model
We also compute the following four individual features for
each primitive object vi:
Area, φ6(vi),
Eccentricity, φ7(vi),
Solidity, φ8(vi),
Regularity, φ9(vi).
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 11 / 78
Compound Structure Model
Spatial Arrangement Model
A one-dimensional marginal histogram Hr1r2
k (Er1r2 ) is
constructed for each pairwise feature φk , k = 1, . . . , 5,
computed over all edges for each pair of layers Vr1 and Vr2 .
Also, a one-dimensional marginal histogram Hr
k (Vr
) is
constructed for each individual feature φk , k = 6, . . . , 8,
computed over all vertices at each layer Vr
.
The concatenation H(V) of all marginal histograms is used
as a non-parametric approximation to the distribution of the
primitive objects in the compound structure.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 12 / 78
Compound Structure Model
Spatial Arrangement Model
Figure: Example histograms for the building layers of four different
types of compound structures.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 13 / 78
Compound Structure Model
Probabilistic Region Processes
Each primitive object vi (i.e., the ellipse parameters) is
considered a vector-valued random variable.
A compound structure is represented by a set of random
variables that leads to a region process.
The region process is governed by the Gibbs distribution
p(V|β) =
1
Zv
exp βT
H(V) (1)
where β is the parameter vector controlling each histogram
bin, and Zv is the partition function.
A region process is equivalent to a Markov random field
(MRF).
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 14 / 78
Learning
Maximum Likelihood Estimation
Suppose that we observe a set of i.i.d. region processes
V = {V1, . . . , VM}.
We can estimate a compound structure model via
maximum likelihood estimation (MLE) of β by maximizing
(β|V) =
M
m=1
log p(Vm|β). (2)
The gradient of the log-likelihood is given by
d (β|V)
dβ
= Ep[H(V)] −
1
M
N
m=1
H(Vm). (3)
We use the stochastic gradient ascent algorithm where the
expectation Ep[H(V)] is approximated by a finite sum of
histograms of samples V(s), s = 1, . . . , S.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 15 / 78
Learning
Maximum Likelihood Estimation
Figure: An example iteration for updating β corresponding to the
relative orientation histogram bins.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 16 / 78
Learning
Sampling Region Processes
(a) (b) t = 0 (c) t = 50
(d) t = 200 (e) t = 600 (f) t = 1000
Figure: Illustration of the Gibbs sampler for two primitive layers.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 17 / 78
Inference and Region Selection
Hierarchical Region Extraction
Given a compound structure model with learned parameter
vector β, we would like to automatically detect all of its
instances in an input image.
The detection problem is posed as the selection of multiple
subgroups of candidate regions coming from multiple
hierarchical segmentations.
Figure: Hierarchical segmentation trees for two primitive layers.
Each selected group of regions constitutes an instance of
the example compound structure in the large image.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 18 / 78
Inference and Region Selection
Hierarchical Region Extraction
Given a compound structure model with learned parameter
vector β, we would like to automatically detect all of its
instances in an input image.
The detection problem is posed as the selection of multiple
subgroups of candidate regions coming from multiple
hierarchical segmentations.
Figure: Hierarchical segmentation trees for two primitive layers.
Each selected group of regions constitutes an instance of
the example compound structure in the large image.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 18 / 78
Inference and Region Selection
Hierarchical Region Extraction
The first step is the identification of candidate regions for
each layer Vr
by using a hierarchical segmentation
algorithm.
The next step is to connect the potentially related vertices
at all levels to represent the neighbor relationships.
Within-level edges (⊆ Er1r2 , r1 = r2): Voronoi tessellations.
Between-level edges (⊆ Er1r2 , r1 = r2): Ancestor-descendant
relations.
Between-layer edges (⊆ Er1r2 , r1 = r2): Proximity-based
neighbors.
Figure: Hierarchical segmentation trees for two primitive layers.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 19 / 78
Inference and Region Selection
Hierarchical Region Extraction
The first step is the identification of candidate regions for
each layer Vr
by using a hierarchical segmentation
algorithm.
The next step is to connect the potentially related vertices
at all levels to represent the neighbor relationships.
Within-level edges (⊆ Er1r2 , r1 = r2): Voronoi tessellations.
Between-level edges (⊆ Er1r2 , r1 = r2): Ancestor-descendant
relations.
Between-layer edges (⊆ Er1r2 , r1 = r2): Proximity-based
neighbors.
Figure: Hierarchical segmentation trees for two primitive layers.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 19 / 78
Inference and Region Selection
Hierarchical Region Extraction
Figure: Graph construction for two primitive layers (i.e., building and
pool). The hierarchical candidate regions at three and two levels for
these layers are shown in red and light blue, respectively. The edges
that represent parent-child relationship for both layers are shown.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 20 / 78
Inference and Region Selection
Hierarchical Region Extraction
Figure: Graph construction for two primitive layers (i.e., building and
pool). The edges that represent the within- and between-level
neighbor relationship within the same layer are shown.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 21 / 78
Inference and Region Selection
Hierarchical Region Extraction
Figure: Graph construction for two primitive layers (i.e., building and
pool). The edges that represent the within- and between-level
neighbor relationship between the layers are shown. For better
visualization of edges, only 20 percent of all between-layer edges are
shown.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 22 / 78
Inference and Region Selection Inference without Constraints
Bayesian Formulation
Given a graph G = (V, E), the problem can be formulated
as the selection of a subset V∗
among all regions V as
V∗
= arg max
V ⊆V
p(V |I) = arg max
V ⊆V
p(I|V )p(V ) (4)
where p(I|V ) is the observed spectral data likelihood for
the compound structure in the image, and p(V ) acts as the
spatial prior according to the learned appearance and
arrangement model.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 23 / 78
Inference and Region Selection Inference without Constraints
CRF Formulation
We formulate the selection problem in (4) using a
conditional random field (CRF).
Let X = {x1, . . . , xM} where xi ∈ {0, 1}, i = 1, . . . , M, be the
set of indicator variables associated with the vertices V of G
so that xi = 1 implies region vi being selected.
Our CRF formulation defines a posterior distribution as
p(X|I, V) ∝ p(I|X, V)p(X, V)
=
1
Zx
vi ∈V
exp ψc
i + ψs
i xi
(vi ,vj )∈E
exp ψa
ij xixj . (5)
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 24 / 78
Inference and Region Selection Inference without Constraints
CRF Formulation
The vertex bias terms ψc
and ψs
representing color and
shape, respectively, and edge weights ψa
representing
arrangement are defined as
ψc
i =
−1
2
(yi − µr
)T
(Σr
)−1
(yi − µr ), ∀vi ∈ Vr
, r = 1, . . . , R
(6)
ψs
i =
9
k=6
βr
k,Ir
k
φk (vi )
, ∀vi ∈ Vr
, r = 1, . . . , R
(7)
ψa
ij =
5
k=1
βr1r2
k,I
r1r2
k
φk (vi ,vj )
, ∀(vi, vj) ∈ E, r1, r2 = 1, . . .
(8)
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 25 / 78
Inference and Region Selection Inference without Constraints
CRF Inference
Selecting V∗
in (4) is equivalent to estimating the joint MAP
labels given by
X∗
= arg max
X
p(X|I, V). (9)
Exact inference of the CRF formulation is intractable in
general graphs.
An approximate solution can be obtained by a Markov chain
Monte Carlo sampler.
We developed a sampling algorithm that samples the labels
of many variables at once.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 26 / 78
Inference and Region Selection Inference without Constraints
CRF Inference
Figure: Illustration of the primitive sampling procedure.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 27 / 78
Inference with Constraints
Our objective is to obtain the maximum probability
estimates of the indicator variables, xi, i = 1, . . . , N
satisfying convex inequality and equality constraints.
We reformulate the problem as quadratic programming
under convex constraints.
The problem in Equation (4) can be rewritten as
V∗
= arg max
V ⊆V
V ⊆Ω
p(V |I) = arg max
V ⊆V
V ⊆Ω
p(I|V )p(V ). (10)
where Ω ∈ RN
is a nonempty polyhedral convex set
determined by a set of constraints.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 28 / 78
Inference with Constraints
Quadratic Programming Formulation
For a V ⊆ V, log p(V |β∗
) can be written as follows
log p(V |β) =
5
k=1
R
r1=1
R
r2=1 (vi ,vj )∈Er1r2
βr1r2
k,I
r1r2
k
φk (vi ,vj )
xixj
+
9
k=6
R
r=1 vi ∈Vr
βr
k,Ir
k
φk (vi )
xi − log ZX .
(11)
where ZX is the partition function.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 29 / 78
Inference with Constraints
Quadratic Programming Formulation
Let W = 5
k=1
R
r1=1
R
r2=1 Wr1r2
k where each Wr1r2
k is an
N × N affinity matrix.
Each element of this matrix is calculated as
Wr1r2
k (i, j) = −βr1r2
k,I
r1r2
k
φk (vi ,vj )
.
Also, let q = 9
k=6
R
r=1 qr
k where each qr
k is an N × 1
potential vector.
Each element of this vector is calculated as
qr
k (i) = βr
k,Ir
k
φk (vi )
.
The problem can be formulated as
minimize
x
− log p(V |β) =
1
2
XT
WX + qT
X + log ZX
subject to X ∈ Ω,
X ∈ {0, 1}.
(12)
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 30 / 78
Inference with Constraints
DC Programming Inference
The problem can be formulated as
minimize
x
− log p(V |β) =
1
2
XT
WX + qT
X + log ZX
subject to X ∈ Ω,
X ∈ {0, 1}.
(13)
First, a linear programming relaxation is applied to the 0 − 1
integer program so that 0 ≤ x ≤ 1.
Since W is not assumed positive semidefinite, the resulting
linearly constrained quadratic problem is not convex.
The objective function can be reformulated as a difference
of two convex functions.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 31 / 78
Inference with Constraints
Difference of Convex Programming
Formulation
A Difference of Convex (DC) problem is defined as
P = min f(X) = g(X) − h(X) : X ∈ RN
(14)
where g : RN
→ R and h : RN
→ R are convex functions.
Consider the dual program
D = min f∗
(Y) = h∗
(Y) − g∗
(Y) : Y ∈ RN
(15)
where g∗
is the conjugate function of g.
An iterative primal-dual algorithm constructs two alternating
sequences {X(t)
} and {Y(t)
} such that
g(X(t)) − h(X(t)) and g∗(Y(t)) − h∗(Y(t)) are decreasing,
converging to the optimal solutions, X∗ and Y∗, to the primal
and dual problems, respectively.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 32 / 78
Inference with Constraints
DC Programming Inference
Let W = QΛQT
be the eigenvalue decomposition of W, and
Λ+
= diag(Λ+
1 , . . . , Λ+
N ) (respectively, Λ−
= diag(Λ−
1 , . . . , Λ−
N ))
be the positive semidefinite diagonal matrix (respectively,
negative semidefinite diagonal matrix) of Λ.
We rewrite the nonconvex symmetric quadratic objective
function as
1
2
XT
WX + qT
X = g(X) − h(X)
g(X) =
1
2
XT
W+
X + qT
X + χΩ(X)
h(X) = −
1
2
XT
W−
X
(16)
where W+
= QΛ+
QT
, W−
= QΛ−
QT
, and χΩ(X) is the
space enclosed by the constraints X ∈ {0, 1} and Ω.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 33 / 78
Inference with Constraints
Experiments
We present detailed results of four different kinds of
experiments:
1 Using a single layer without imposing any constraint.
2 Using multiple layers without imposing any constraint.
3 Using a single layer by imposing constraints.
4 Using multiple layers by imposing constraints.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 34 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Figure: 2500 × 4000 pixels Ankara data set.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 35 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Table: Detection scenarios for the experiments. Example primitives
used for learning the compound structure model for each scenario are
shown in a different color. The number of polygons and buildings in
the validation data are also given.
Scenario 1 2 3 4 5 6
Example
primitives
# polygons 162 98 48 195 60 16
# buildings 1519 870 1117 1796 771 219
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 36 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Figure: Candidate regions hierarchy.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 37 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Figure: Marginal probabilities for the first scenario.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 38 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Figure: Marginal probabilities for the second scenario.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 39 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Figure: Marginal probabilities for the third scenario.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 40 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Figure: Marginal probabilities for the fourth scenario.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 41 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Figure: Marginal probabilities for the fifth scenario.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 42 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
*
Figure: Marginal probabilities for the sixth scenario.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 43 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Figure: Zoomed detection examples.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 44 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Figure: Zoomed detection examples.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 45 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Figure: Zoomed detection examples.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 46 / 78
Experiments Unconstrained Single-layer Experiments
Results-Urban Structures
Figure: Zoomed detection examples. Each row corresponds to a
particular scenario.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 47 / 78
Experiments Unconstrained Single-layer Experiments
Results-Orchards
Figure: 3000 × 8000 pixels Kusadasi data set.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 48 / 78
Experiments Unconstrained Single-layer Experiments
Results-Orchards
Figure: (Up) Candidate regions. (Down) Selected regions.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 49 / 78
Experiments Unconstrained Single-layer Experiments
Results-Orchards
Figure: Example results for the detection of orchards in the subimage
on the left column. The right column shows the corresponding
marginal probabilities of the selected regions (the copper colormap) as
well as the discarded input candidate regions (white).
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 50 / 78
Experiments Unconstrained Single-layer Experiments
Results-Orchards
Figure: Example results for the detection of orchards in the subimage
on the left column. The right column shows the corresponding
marginal probabilities of the selected regions (the copper colormap) as
well as the discarded input candidate regions (white).
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 51 / 78
Experiments Unconstrained Single-layer Experiments
Results-Orchards
Figure: Example results for the detection of orchards. The left column
shows the marginal probabilities at the end of selection. The right
column shows the thresholded detections overlayed as red.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 52 / 78
Experiments Unconstrained Single-layer Experiments
Results-Refugee Camps
Figure: 1102 × 971 pixels Darfur data set.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 53 / 78
Experiments Unconstrained Single-layer Experiments
Results-Refugee Camps
Figure: Example results for the detection of refugee camps as rural
structures.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 54 / 78
Experiments Unconstrained Multi-layer Experiments
Results-Housing Estates
Figure: 3000 × 8000 pixels Kusadasi data set.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 55 / 78
Experiments Unconstrained Multi-layer Experiments
Results-Housing Estates
Figure: (Up) Examples of local details of red building rooftops. (Down)
An example hierarchy.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 56 / 78
Experiments Unconstrained Multi-layer Experiments
Results-Housing Estates
The selection algorithm that used only the building layer
could not detect several housing estates.
The idea was to add a pool layer that can provide additional
cues for finding the missed buildings.
The initial model was extended by learning the
arrangements of buildings with respect to pools as well.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 57 / 78
Experiments Unconstrained Multi-layer Experiments
Results-Housing Estates
Figure: (Up) Candidate regions. (Down) Selected regions from the
building and pool layers.H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 58 / 78
Experiments Unconstrained Multi-layer Experiments
Results-Housing Estates
Table: The number of candidate and detected regions for single and
multi-layer selection scenarios.
Single-layer Multi-layer
Candidates Detected Candidates Detected
Building 67,983 11,173 67,983 11,871
Pool - - 16,276 436
Total 67,983 11,173 84,259 12,307
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 59 / 78
Experiments Unconstrained Multi-layer Experiments
Results-Housing Estates
Figure: Samples obtained by the selection procedure ran on single
and multiple layers.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 60 / 78
Experiments Unconstrained Multi-layer Experiments
Results-Housing Estates
(a) (b)
Figure: The selected regions using (a) only the building layer. (b)
building and pool layers. Newly detected housing estates that was
missed with single layer selection is enclosed by a red convex hull.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 61 / 78
Experiments Unconstrained Multi-layer Experiments
Results-Housing Estates
(a) (b)
Figure: The selected regions using (a) only the building layer. (b)
building and pool layers. Newly detected housing estates that was
missed with single layer selection is enclosed by a red convex hull.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 62 / 78
Experiments Unconstrained Multi-layer Experiments
Results-Housing Estates
Figure: Ground view of a missed housing estate with single layer
selection.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 63 / 78
Experiments
Constrained Single-layer Experiments
Figure: Zoomed detection examples.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 64 / 78
Experiments
Problem Definition
Unconstrained selection involved overlapping regions at
different levels of the hierarchy.
To overcome this problem, we require at most one region
should be selected per path where a path corresponds to
the set of vertices from a leaf to the root.
Formally, we select an optimal subset V∗
⊆ V such that
∀a, b ∈ V∗
, a ∈ descendant(b) and b ∈ descendant(a).
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 65 / 78
Experiments
Constrained Single-layer Experiments
Let A be a |P| × |V| matrix.
P denotes all the paths from the leaves to the roots of the
input hierarchical forest.
A(i, j) = 1 implies vi ∈ pj ∈ P.
The problem can be reformulated as
minimize
x
1
2
XT
WX + qT
X + log ZX
subject to AX ≤ 1,
0 ≤ X ≤ 1.
(17)
The resulting problem is solved by the DC inference
algorithm.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 66 / 78
Experiments
Results-Urban Structures
Table: The number of selected regions for unconstrained and
constrained selection scenarios.
# cand.s 70,644 70,644 70,644 70,644 70,644 22,195
Uncnstr. 3191 1828 3819 3201 2027 1612
Cnstr. 1485 856 2562 1740 811 263
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 67 / 78
Experiments
Results-Urban Structures
(a) (b)
Figure: Zoomed detection examples. (a) shows the RGB image for a
300 × 300 sub-scene. (b) shows the hierarchy of candidate regions
(two-level hierarchy from bottom to top).
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 68 / 78
Experiments
Results-Urban Structures
(a)
(b)
Figure: Zoomed detection examples. (a) shows the RGB image for a
300 × 300 sub-scene. (b) shows the hierarchy of candidate regions
(six-level hierarchy from bottom to top).
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 69 / 78
Experiments
Constrained Multi-layer Experiments
The last set of experiments uses two primitive layers and
enforces geometrical constraints between them.
We search for nearby alike buildings and green areas where
each building group must have a green area in the middle.
We strictly require that the distance between the centroid of
the centroids of a selected group of similar buildings and
the centroid of a selected large green area cannot exceed a
distance threshold δ .
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 70 / 78
Experiments
Constrained Multi-layer Experiments
Let V1
and V2
represent the building and green area layers.
The desired set of regions can be selected by
minimize
x
1
2
XT
WX + qT
X + log ZX
subject to AX ≤ 1,
1
k1
vi ∈V1
sh
i xi −
1
k2
vj ∈V2
sh
j xj ≤ δ
1
k1
vi ∈V1
sw
i xi −
1
k2
vj ∈V2
sw
j xj ≤ δ
vi ∈V1
xi = k1
vj ∈V2
xj = k2
0 ≤ X ≤ 1.
(18)
where (sh
i , sw
i ) is the centroid of region vi, k1 and k2 denote
the number of buildings and green areas to be selected.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 71 / 78
Experiments Constrained Multi-layer Experiments
Results-Buildings & Green Areas
Figure: 2500 × 4000 pixels Ankara data set.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 72 / 78
Experiments Constrained Multi-layer Experiments
Results-Buildings & Green Areas
Figure: Selected regions for the green areas surrounded by buildings.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 73 / 78
Experiments Constrained Multi-layer Experiments
Results-Buildings & Green Areas
(a) RGB (b) Building candidates
(c) Green candi-
dates
(d) Selection (e) Overlay
Figure: A zoomed detection example for k1 = 4, k2 = 1.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 74 / 78
Experiments Constrained Multi-layer Experiments
Results-Buildings & Green Areas
(a) RGB (b) Building candidates
(c) Green candi-
dates
(d) Selection (e) Overlay
Figure: A zoomed detection example for k1 = 6, k2 = 1.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 75 / 78
Experiments Constrained Multi-layer Experiments
Results-Buildings & Green Areas
(a) k1 = 4, fVal = −2.95 (b) k1 = 8, fVal = −3.23
(c) k1 = 6, fVal = −3.16 (d) k1 = 4, fVal = −2.97
Figure: Zoomed detection examples for different values of k1 = 4, 6, 8.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 76 / 78
Experiments
Summary & Conclusions
We described a generic method for the modeling and
detection of compound structures that consisted of
arrangements of mostly unknown number of primitives.
The modeling process built an MRF-based contextual
model for the compound structure of interest.
The detection task involved a combinatorial selection
problem where multiple subsets of candidate regions from
multiple hierarchical segmentations were selected.
We also handled hard constraints on the candidate regions.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 77 / 78
Experiments
Summary & Conclusions
Experiments using urban, industrial, agricultural and rural
structures showed that the proposed method can provide
good localization of instances of compound structures.
The multi-layered experiments showed that selection of
some objects required the selection of objects in other
layers that had spatial relation with them.
One of the most important bottlenecks in terms of accuracy
was the errors in the input hierarchical segmentations.
Future work includes
Using the detection results for adjusting wrong segmentation
results.
Inferring the primitive objects inside a compound structure.
H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 78 / 78

More Related Content

What's hot

B05531119
B05531119B05531119
B05531119
IOSR-JEN
 
Применение машинного обучения для навигации и управления роботами
Применение машинного обучения для навигации и управления роботамиПрименение машинного обучения для навигации и управления роботами
Применение машинного обучения для навигации и управления роботами
Skolkovo Robotics Center
 
Tl
TlTl
07 a70110 remotesensingandgisapplications
07 a70110 remotesensingandgisapplications07 a70110 remotesensingandgisapplications
07 a70110 remotesensingandgisapplications
imaduddin91
 
Spme 2013 segmentation
Spme 2013 segmentationSpme 2013 segmentation
Spme 2013 segmentation
Qujiang Lei
 
Petrel course Module_1: Import data and management, make simple surfaces
Petrel course Module_1: Import data and management, make simple surfacesPetrel course Module_1: Import data and management, make simple surfaces
Petrel course Module_1: Import data and management, make simple surfaces
Marc Diviu Franco
 
Petrel introduction course guide
Petrel introduction course guidePetrel introduction course guide
Petrel introduction course guide
Marc Diviu Franco
 
Mahaviracharya (hindi)
Mahaviracharya (hindi)Mahaviracharya (hindi)
Mahaviracharya (hindi)
dgupta330
 
final_presentation
final_presentationfinal_presentation
final_presentation
Chiraz Nafouki
 
4-CONNECTED AND 8-CONNECTED NEIGHBOR SELECTION By Sintiak Haque
4-CONNECTED AND 8-CONNECTED NEIGHBOR SELECTION  By Sintiak Haque4-CONNECTED AND 8-CONNECTED NEIGHBOR SELECTION  By Sintiak Haque
4-CONNECTED AND 8-CONNECTED NEIGHBOR SELECTION By Sintiak Haque
Sintiak haque
 
Remotesensingandgisapplications
RemotesensingandgisapplicationsRemotesensingandgisapplications
Remotesensingandgisapplications
imaduddin91
 
User guide of reservoir geological modeling v2.2.0
User guide of reservoir geological modeling v2.2.0User guide of reservoir geological modeling v2.2.0
User guide of reservoir geological modeling v2.2.0
Bo Sun
 
위성이미지 객체 검출 대회 - 2등
위성이미지 객체 검출 대회 - 2등위성이미지 객체 검출 대회 - 2등
위성이미지 객체 검출 대회 - 2등
DACON AI 데이콘
 
Manual of Remote Sensing
Manual of Remote SensingManual of Remote Sensing
Generalized Low Rank Models
Generalized Low Rank ModelsGeneralized Low Rank Models
Generalized Low Rank Models
Sri Ambati
 
PowerLyra@EuroSys2015
PowerLyra@EuroSys2015PowerLyra@EuroSys2015
PowerLyra@EuroSys2015
realstolz
 
H2O World - GLRM - Anqi Fu
H2O World - GLRM - Anqi FuH2O World - GLRM - Anqi Fu
H2O World - GLRM - Anqi Fu
Sri Ambati
 
Highs and Lows: A Resel-based Approach to the Analysis of Data from Geophysic...
Highs and Lows: A Resel-based Approach to the Analysis of Data from Geophysic...Highs and Lows: A Resel-based Approach to the Analysis of Data from Geophysic...
Highs and Lows: A Resel-based Approach to the Analysis of Data from Geophysic...
Jessica Ogden
 
H2O World - Generalized Low Rank Models - Madeleine Udell
H2O World - Generalized Low Rank Models - Madeleine UdellH2O World - Generalized Low Rank Models - Madeleine Udell
H2O World - Generalized Low Rank Models - Madeleine Udell
Sri Ambati
 
ILWIS 3.0 SUMMARY PRESENTATION
ILWIS 3.0 SUMMARY PRESENTATIONILWIS 3.0 SUMMARY PRESENTATION
ILWIS 3.0 SUMMARY PRESENTATION
musadoto
 

What's hot (20)

B05531119
B05531119B05531119
B05531119
 
Применение машинного обучения для навигации и управления роботами
Применение машинного обучения для навигации и управления роботамиПрименение машинного обучения для навигации и управления роботами
Применение машинного обучения для навигации и управления роботами
 
Tl
TlTl
Tl
 
07 a70110 remotesensingandgisapplications
07 a70110 remotesensingandgisapplications07 a70110 remotesensingandgisapplications
07 a70110 remotesensingandgisapplications
 
Spme 2013 segmentation
Spme 2013 segmentationSpme 2013 segmentation
Spme 2013 segmentation
 
Petrel course Module_1: Import data and management, make simple surfaces
Petrel course Module_1: Import data and management, make simple surfacesPetrel course Module_1: Import data and management, make simple surfaces
Petrel course Module_1: Import data and management, make simple surfaces
 
Petrel introduction course guide
Petrel introduction course guidePetrel introduction course guide
Petrel introduction course guide
 
Mahaviracharya (hindi)
Mahaviracharya (hindi)Mahaviracharya (hindi)
Mahaviracharya (hindi)
 
final_presentation
final_presentationfinal_presentation
final_presentation
 
4-CONNECTED AND 8-CONNECTED NEIGHBOR SELECTION By Sintiak Haque
4-CONNECTED AND 8-CONNECTED NEIGHBOR SELECTION  By Sintiak Haque4-CONNECTED AND 8-CONNECTED NEIGHBOR SELECTION  By Sintiak Haque
4-CONNECTED AND 8-CONNECTED NEIGHBOR SELECTION By Sintiak Haque
 
Remotesensingandgisapplications
RemotesensingandgisapplicationsRemotesensingandgisapplications
Remotesensingandgisapplications
 
User guide of reservoir geological modeling v2.2.0
User guide of reservoir geological modeling v2.2.0User guide of reservoir geological modeling v2.2.0
User guide of reservoir geological modeling v2.2.0
 
위성이미지 객체 검출 대회 - 2등
위성이미지 객체 검출 대회 - 2등위성이미지 객체 검출 대회 - 2등
위성이미지 객체 검출 대회 - 2등
 
Manual of Remote Sensing
Manual of Remote SensingManual of Remote Sensing
Manual of Remote Sensing
 
Generalized Low Rank Models
Generalized Low Rank ModelsGeneralized Low Rank Models
Generalized Low Rank Models
 
PowerLyra@EuroSys2015
PowerLyra@EuroSys2015PowerLyra@EuroSys2015
PowerLyra@EuroSys2015
 
H2O World - GLRM - Anqi Fu
H2O World - GLRM - Anqi FuH2O World - GLRM - Anqi Fu
H2O World - GLRM - Anqi Fu
 
Highs and Lows: A Resel-based Approach to the Analysis of Data from Geophysic...
Highs and Lows: A Resel-based Approach to the Analysis of Data from Geophysic...Highs and Lows: A Resel-based Approach to the Analysis of Data from Geophysic...
Highs and Lows: A Resel-based Approach to the Analysis of Data from Geophysic...
 
H2O World - Generalized Low Rank Models - Madeleine Udell
H2O World - Generalized Low Rank Models - Madeleine UdellH2O World - Generalized Low Rank Models - Madeleine Udell
H2O World - Generalized Low Rank Models - Madeleine Udell
 
ILWIS 3.0 SUMMARY PRESENTATION
ILWIS 3.0 SUMMARY PRESENTATIONILWIS 3.0 SUMMARY PRESENTATION
ILWIS 3.0 SUMMARY PRESENTATION
 

Viewers also liked

Hoja informativa sobre el cáncer de cuello Utero
Hoja informativa sobre el cáncer de cuello UteroHoja informativa sobre el cáncer de cuello Utero
Hoja informativa sobre el cáncer de cuello Utero
Pablo Benavides
 
Design for People, Effective Innovation and Sustainability
Design for People, Effective Innovation and SustainabilityDesign for People, Effective Innovation and Sustainability
Design for People, Effective Innovation and Sustainability
Musstanser Tinauli
 
Design thiking e gestão de projetos - INFOBRAL 2013
Design thiking e gestão de projetos   - INFOBRAL 2013Design thiking e gestão de projetos   - INFOBRAL 2013
Design thiking e gestão de projetos - INFOBRAL 2013
Eduardo Freire
 
PhD Thesis Defense Presentation - Estudo da viabilidade de fabricação de disp...
PhD Thesis Defense Presentation - Estudo da viabilidade de fabricação de disp...PhD Thesis Defense Presentation - Estudo da viabilidade de fabricação de disp...
PhD Thesis Defense Presentation - Estudo da viabilidade de fabricação de disp...
Alessandro Oliveira
 
Phd. presentation khaled asaiah
Phd. presentation khaled asaiahPhd. presentation khaled asaiah
Phd. presentation khaled asaiah
khaled asaiah
 
PALESTRA - Inovação em Gerenciamento de Projetos - Eduardo Freire
PALESTRA - Inovação em Gerenciamento de Projetos - Eduardo FreirePALESTRA - Inovação em Gerenciamento de Projetos - Eduardo Freire
PALESTRA - Inovação em Gerenciamento de Projetos - Eduardo Freire
Papo de Consultor
 
Deploma
DeplomaDeploma
سرطان الثدي Final
سرطان الثدي Finalسرطان الثدي Final
سرطان الثدي Final
Sara Salem
 
Easy Tips for Public Speaking | Raychel Harvey Jones
Easy Tips for Public Speaking |  Raychel Harvey JonesEasy Tips for Public Speaking |  Raychel Harvey Jones
Easy Tips for Public Speaking | Raychel Harvey Jones
Raychel Harvey Jones
 
A PhD: A Career in Nursing Research
A PhD: A Career in Nursing ResearchA PhD: A Career in Nursing Research
A PhD: A Career in Nursing Research
Kelly Brittain
 
WIUGC 2010 - Pegmatite and Leucogranite-Hosted U-Th Mineralization In norther...
WIUGC 2010 - Pegmatite and Leucogranite-Hosted U-Th Mineralization In norther...WIUGC 2010 - Pegmatite and Leucogranite-Hosted U-Th Mineralization In norther...
WIUGC 2010 - Pegmatite and Leucogranite-Hosted U-Th Mineralization In norther...
Christine McKechnie
 
ApresentaçãO Tese Doutoramento
ApresentaçãO Tese DoutoramentoApresentaçãO Tese Doutoramento
ApresentaçãO Tese Doutoramento
Neuza Pedro
 
Islamisation
IslamisationIslamisation
IslamisationToorop
 
Reformation or Islamisation
Reformation or IslamisationReformation or Islamisation
Reformation or Islamisation
Peter Hammond
 
The Islamisation of Europe - What can be Done to Stop and Reverse It
The Islamisation of Europe - What can be Done to Stop and Reverse ItThe Islamisation of Europe - What can be Done to Stop and Reverse It
The Islamisation of Europe - What can be Done to Stop and Reverse It
Peter Hammond
 
Thesis presentation_ Final Defence
Thesis presentation_ Final DefenceThesis presentation_ Final Defence
Thesis presentation_ Final Defence
Sharmin Ahmed
 
Thesis defence
Thesis defence Thesis defence
Thesis defence
Mr.akkaluk Moppijit
 
Islamization of knowledge: Special Reference to the Discipline of Fiqh and Us...
Islamization of knowledge: Special Reference to the Discipline of Fiqh and Us...Islamization of knowledge: Special Reference to the Discipline of Fiqh and Us...
Islamization of knowledge: Special Reference to the Discipline of Fiqh and Us...
Abu Talib Mohammad Monawer
 
Different Tools to Detect and Monitor Oil Spills Aerial Observation Tech.
Different Tools to Detect and Monitor Oil Spills Aerial Observation Tech.Different Tools to Detect and Monitor Oil Spills Aerial Observation Tech.
Different Tools to Detect and Monitor Oil Spills Aerial Observation Tech.
A.Tuğsan İşiaçık Çolak
 
The ePortfolio: The work of an individual?
The ePortfolio: The work of an individual?The ePortfolio: The work of an individual?
The ePortfolio: The work of an individual?
Kristina D.C. Hoeppner
 

Viewers also liked (20)

Hoja informativa sobre el cáncer de cuello Utero
Hoja informativa sobre el cáncer de cuello UteroHoja informativa sobre el cáncer de cuello Utero
Hoja informativa sobre el cáncer de cuello Utero
 
Design for People, Effective Innovation and Sustainability
Design for People, Effective Innovation and SustainabilityDesign for People, Effective Innovation and Sustainability
Design for People, Effective Innovation and Sustainability
 
Design thiking e gestão de projetos - INFOBRAL 2013
Design thiking e gestão de projetos   - INFOBRAL 2013Design thiking e gestão de projetos   - INFOBRAL 2013
Design thiking e gestão de projetos - INFOBRAL 2013
 
PhD Thesis Defense Presentation - Estudo da viabilidade de fabricação de disp...
PhD Thesis Defense Presentation - Estudo da viabilidade de fabricação de disp...PhD Thesis Defense Presentation - Estudo da viabilidade de fabricação de disp...
PhD Thesis Defense Presentation - Estudo da viabilidade de fabricação de disp...
 
Phd. presentation khaled asaiah
Phd. presentation khaled asaiahPhd. presentation khaled asaiah
Phd. presentation khaled asaiah
 
PALESTRA - Inovação em Gerenciamento de Projetos - Eduardo Freire
PALESTRA - Inovação em Gerenciamento de Projetos - Eduardo FreirePALESTRA - Inovação em Gerenciamento de Projetos - Eduardo Freire
PALESTRA - Inovação em Gerenciamento de Projetos - Eduardo Freire
 
Deploma
DeplomaDeploma
Deploma
 
سرطان الثدي Final
سرطان الثدي Finalسرطان الثدي Final
سرطان الثدي Final
 
Easy Tips for Public Speaking | Raychel Harvey Jones
Easy Tips for Public Speaking |  Raychel Harvey JonesEasy Tips for Public Speaking |  Raychel Harvey Jones
Easy Tips for Public Speaking | Raychel Harvey Jones
 
A PhD: A Career in Nursing Research
A PhD: A Career in Nursing ResearchA PhD: A Career in Nursing Research
A PhD: A Career in Nursing Research
 
WIUGC 2010 - Pegmatite and Leucogranite-Hosted U-Th Mineralization In norther...
WIUGC 2010 - Pegmatite and Leucogranite-Hosted U-Th Mineralization In norther...WIUGC 2010 - Pegmatite and Leucogranite-Hosted U-Th Mineralization In norther...
WIUGC 2010 - Pegmatite and Leucogranite-Hosted U-Th Mineralization In norther...
 
ApresentaçãO Tese Doutoramento
ApresentaçãO Tese DoutoramentoApresentaçãO Tese Doutoramento
ApresentaçãO Tese Doutoramento
 
Islamisation
IslamisationIslamisation
Islamisation
 
Reformation or Islamisation
Reformation or IslamisationReformation or Islamisation
Reformation or Islamisation
 
The Islamisation of Europe - What can be Done to Stop and Reverse It
The Islamisation of Europe - What can be Done to Stop and Reverse ItThe Islamisation of Europe - What can be Done to Stop and Reverse It
The Islamisation of Europe - What can be Done to Stop and Reverse It
 
Thesis presentation_ Final Defence
Thesis presentation_ Final DefenceThesis presentation_ Final Defence
Thesis presentation_ Final Defence
 
Thesis defence
Thesis defence Thesis defence
Thesis defence
 
Islamization of knowledge: Special Reference to the Discipline of Fiqh and Us...
Islamization of knowledge: Special Reference to the Discipline of Fiqh and Us...Islamization of knowledge: Special Reference to the Discipline of Fiqh and Us...
Islamization of knowledge: Special Reference to the Discipline of Fiqh and Us...
 
Different Tools to Detect and Monitor Oil Spills Aerial Observation Tech.
Different Tools to Detect and Monitor Oil Spills Aerial Observation Tech.Different Tools to Detect and Monitor Oil Spills Aerial Observation Tech.
Different Tools to Detect and Monitor Oil Spills Aerial Observation Tech.
 
The ePortfolio: The work of an individual?
The ePortfolio: The work of an individual?The ePortfolio: The work of an individual?
The ePortfolio: The work of an individual?
 

Similar to Compound Structure Detection

International Journal of Engineering Research and Development
International Journal of Engineering Research and DevelopmentInternational Journal of Engineering Research and Development
International Journal of Engineering Research and Development
IJERD Editor
 
Projection methods for stochastic structural dynamics
Projection methods for stochastic structural dynamicsProjection methods for stochastic structural dynamics
Projection methods for stochastic structural dynamics
University of Glasgow
 
DETECTION OF MOVING OBJECT
DETECTION OF MOVING OBJECTDETECTION OF MOVING OBJECT
DETECTION OF MOVING OBJECT
AM Publications
 
SPSF02 - Graphical Data Representation
SPSF02 - Graphical Data RepresentationSPSF02 - Graphical Data Representation
SPSF02 - Graphical Data Representation
Syeilendra Pramuditya
 
Steven Duplij, Raimund Vogl, "Polyadic Braid Operators and Higher Braiding Ga...
Steven Duplij, Raimund Vogl, "Polyadic Braid Operators and Higher Braiding Ga...Steven Duplij, Raimund Vogl, "Polyadic Braid Operators and Higher Braiding Ga...
Steven Duplij, Raimund Vogl, "Polyadic Braid Operators and Higher Braiding Ga...
Steven Duplij (Stepan Douplii)
 
VEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVMVEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVM
cseij
 
VEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVMVEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVM
cseij
 
VEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVM VEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVM
cseij
 
Vehicle Recognition Using VIBE and SVM
Vehicle Recognition Using VIBE and SVMVehicle Recognition Using VIBE and SVM
Vehicle Recognition Using VIBE and SVM
CSEIJJournal
 
Instance Based Learning in Machine Learning
Instance Based Learning in Machine LearningInstance Based Learning in Machine Learning
Instance Based Learning in Machine Learning
Pavithra Thippanaik
 
Mask R-CNN
Mask R-CNNMask R-CNN
Mask R-CNN
Chanuk Lim
 
Hierarchical Self Attention Based Autoencoder for Open-Set Human Activity Rec...
Hierarchical Self Attention Based Autoencoder for Open-Set Human Activity Rec...Hierarchical Self Attention Based Autoencoder for Open-Set Human Activity Rec...
Hierarchical Self Attention Based Autoencoder for Open-Set Human Activity Rec...
Saif Mahmud
 
Convolutional networks and graph networks through kernels
Convolutional networks and graph networks through kernelsConvolutional networks and graph networks through kernels
Convolutional networks and graph networks through kernels
tuxette
 
BALANCING BOARD MACHINES
BALANCING BOARD MACHINESBALANCING BOARD MACHINES
BALANCING BOARD MACHINES
butest
 
AbdoSummerANS_mod3
AbdoSummerANS_mod3AbdoSummerANS_mod3
AbdoSummerANS_mod3
Mohammad Abdo
 
Prpagation of Error Bounds Across reduction interfaces
Prpagation of Error Bounds Across reduction interfacesPrpagation of Error Bounds Across reduction interfaces
Prpagation of Error Bounds Across reduction interfaces
Mohammad
 
VoxelNet
VoxelNetVoxelNet
VoxelNet
taeseon ryu
 
Entropy 19-00079
Entropy 19-00079Entropy 19-00079
Entropy 19-00079
Mazharul Islam
 
Background Estimation Using Principal Component Analysis Based on Limited Mem...
Background Estimation Using Principal Component Analysis Based on Limited Mem...Background Estimation Using Principal Component Analysis Based on Limited Mem...
Background Estimation Using Principal Component Analysis Based on Limited Mem...
IJECEIAES
 
AU QP Answer key NOv/Dec 2015 Computer Graphics 5 sem CSE
AU QP Answer key NOv/Dec 2015 Computer Graphics 5 sem CSEAU QP Answer key NOv/Dec 2015 Computer Graphics 5 sem CSE
AU QP Answer key NOv/Dec 2015 Computer Graphics 5 sem CSE
Thiyagarajan G
 

Similar to Compound Structure Detection (20)

International Journal of Engineering Research and Development
International Journal of Engineering Research and DevelopmentInternational Journal of Engineering Research and Development
International Journal of Engineering Research and Development
 
Projection methods for stochastic structural dynamics
Projection methods for stochastic structural dynamicsProjection methods for stochastic structural dynamics
Projection methods for stochastic structural dynamics
 
DETECTION OF MOVING OBJECT
DETECTION OF MOVING OBJECTDETECTION OF MOVING OBJECT
DETECTION OF MOVING OBJECT
 
SPSF02 - Graphical Data Representation
SPSF02 - Graphical Data RepresentationSPSF02 - Graphical Data Representation
SPSF02 - Graphical Data Representation
 
Steven Duplij, Raimund Vogl, "Polyadic Braid Operators and Higher Braiding Ga...
Steven Duplij, Raimund Vogl, "Polyadic Braid Operators and Higher Braiding Ga...Steven Duplij, Raimund Vogl, "Polyadic Braid Operators and Higher Braiding Ga...
Steven Duplij, Raimund Vogl, "Polyadic Braid Operators and Higher Braiding Ga...
 
VEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVMVEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVM
 
VEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVMVEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVM
 
VEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVM VEHICLE RECOGNITION USING VIBE AND SVM
VEHICLE RECOGNITION USING VIBE AND SVM
 
Vehicle Recognition Using VIBE and SVM
Vehicle Recognition Using VIBE and SVMVehicle Recognition Using VIBE and SVM
Vehicle Recognition Using VIBE and SVM
 
Instance Based Learning in Machine Learning
Instance Based Learning in Machine LearningInstance Based Learning in Machine Learning
Instance Based Learning in Machine Learning
 
Mask R-CNN
Mask R-CNNMask R-CNN
Mask R-CNN
 
Hierarchical Self Attention Based Autoencoder for Open-Set Human Activity Rec...
Hierarchical Self Attention Based Autoencoder for Open-Set Human Activity Rec...Hierarchical Self Attention Based Autoencoder for Open-Set Human Activity Rec...
Hierarchical Self Attention Based Autoencoder for Open-Set Human Activity Rec...
 
Convolutional networks and graph networks through kernels
Convolutional networks and graph networks through kernelsConvolutional networks and graph networks through kernels
Convolutional networks and graph networks through kernels
 
BALANCING BOARD MACHINES
BALANCING BOARD MACHINESBALANCING BOARD MACHINES
BALANCING BOARD MACHINES
 
AbdoSummerANS_mod3
AbdoSummerANS_mod3AbdoSummerANS_mod3
AbdoSummerANS_mod3
 
Prpagation of Error Bounds Across reduction interfaces
Prpagation of Error Bounds Across reduction interfacesPrpagation of Error Bounds Across reduction interfaces
Prpagation of Error Bounds Across reduction interfaces
 
VoxelNet
VoxelNetVoxelNet
VoxelNet
 
Entropy 19-00079
Entropy 19-00079Entropy 19-00079
Entropy 19-00079
 
Background Estimation Using Principal Component Analysis Based on Limited Mem...
Background Estimation Using Principal Component Analysis Based on Limited Mem...Background Estimation Using Principal Component Analysis Based on Limited Mem...
Background Estimation Using Principal Component Analysis Based on Limited Mem...
 
AU QP Answer key NOv/Dec 2015 Computer Graphics 5 sem CSE
AU QP Answer key NOv/Dec 2015 Computer Graphics 5 sem CSEAU QP Answer key NOv/Dec 2015 Computer Graphics 5 sem CSE
AU QP Answer key NOv/Dec 2015 Computer Graphics 5 sem CSE
 

Recently uploaded

06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Timothy Spann
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
Social Samosa
 
My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.
rwarrenll
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
manishkhaire30
 
Global Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headedGlobal Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headed
vikram sood
 
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
nuttdpt
 
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdfUdemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
Fernanda Palhano
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
aqzctr7x
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
Sachin Paul
 
State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023
kuntobimo2016
 
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
Walaa Eldin Moustafa
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
74nqk8xf
 
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
AlessioFois2
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
Timothy Spann
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
Sm321
 
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Aggregage
 
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
zsjl4mimo
 
Influence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business PlanInfluence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business Plan
jerlynmaetalle
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Kiwi Creative
 
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
sameer shah
 

Recently uploaded (20)

06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
 
My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
 
Global Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headedGlobal Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headed
 
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
 
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdfUdemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
Udemy_2024_Global_Learning_Skills_Trends_Report (1).pdf
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
 
State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023
 
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
 
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
 
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
 
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
 
Influence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business PlanInfluence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business Plan
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
 
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
 

Compound Structure Detection

  • 1. AUTOMATIC DETECTION OF COMPOUND STRUCTURES FROM MULTIPLE HIERARCHICAL SEGMENTATIONS H¨useyin G¨okhan Akc¸ay Department of Computer Engineering, Bilkent University, Bilkent, 06800, Ankara akcay@cs.bilkent.edu.tr 21 Sept. 2016 H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 1 / 78
  • 2. Motivation Large scale global content about the Earth. Small local details (upto 30 cm resolution). 1.6 terabytes of data by the ESA’s multispectral high-resolution imaging satellite. The WorldView-2 satellite collects 975,000 square kilometers of imagery per day. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 2 / 78
  • 3. Motivation A challenging problem in remote sensing image mining is the detection of heterogeneous compound structures such as different types of residential, industrial, and agricultural areas. Compound structures are comprised of spatial arrangements of simple primitive objects such as buildings, trees and road segments. Detection of compound structures is a challenging problem because They contain thousands of primitive objects. They mostly do not have distinctive features. Primitives can arrange in many different combinations in the overhead view. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 3 / 78
  • 4. Motivation Figure: 75 × 75m2 compound structures in WorldView-2 images. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 4 / 78
  • 5. Literature Review Primitive object detection. Residential-factory buildings, local roads, vehicles, airplanes and boats. Window-based approaches. Bag-of-words representation. Enforces artificial boundaries on the image. Assumes the whole window corresponds to a compound structure. Segmentation-based approaches. The grouping criteria do not involve spatial arrangements. Graph-based approaches. Specific arrangements such as alignment and parallelism. Structural graph matching. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 5 / 78
  • 6. Problem Definition We propose a generic method for the modeling and detection of compound structures. Target structures can involve arrangements of an unknown number of different types of primitive objects. The detection task is formulated as the selection of multiple coherent subsets of candidate regions obtained from multiple hierarchical segmentations. To avoid over- or under-segmentation of candidate regions, we search for the most meaningful regions at different scales. We propose a constrained region selection framework which allows to specify global constraints on the selected regions. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 6 / 78
  • 8. Compound Structure Model Primitive Representation Compound structures are composed of spatial arrangements of multiple, relatively homogeneous, and compact primitive objects. We assume that a compound structure V consists of R layers of primitive object maps, V = r=1,...,R Vr . Figure: Primitive object layers. Each primitive object vi is represented by an ellipse vi = (li, si, θi). H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 8 / 78
  • 9. Compound Structure Model Spatial Arrangement Model For a given compound structure consisting of N primitive objects, we construct a neighborhood graph G = (V, E). V = {v1, . . . , vN} correspond to the individual primitive objects, E = r1,r2=1,...,R Er1r2 where Er1r2 denotes the edges between the vertices at layers Vr1 and Vr2 . Figure: Neighborhood graph construction for multiple primitive layers. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 9 / 78
  • 10. Compound Structure Model Spatial Arrangement Model For each (vi, vj) ∈ E, we compute the following five features: Distance between the closest pixels, φ1(vi, vj), Relative orientation, φ2(vi, vj), φ2 φ3 φ1 φ4 Angle between the line joining the centroids of the two objects and the major axis of vi as the reference object, φ3(vi, vj), Distance between the closest antipodal pixels that lie on the major axes, φ4(vi, vj), Relative size, φ5(vi, vj). H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 10 / 78
  • 11. Compound Structure Model Spatial Arrangement Model We also compute the following four individual features for each primitive object vi: Area, φ6(vi), Eccentricity, φ7(vi), Solidity, φ8(vi), Regularity, φ9(vi). H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 11 / 78
  • 12. Compound Structure Model Spatial Arrangement Model A one-dimensional marginal histogram Hr1r2 k (Er1r2 ) is constructed for each pairwise feature φk , k = 1, . . . , 5, computed over all edges for each pair of layers Vr1 and Vr2 . Also, a one-dimensional marginal histogram Hr k (Vr ) is constructed for each individual feature φk , k = 6, . . . , 8, computed over all vertices at each layer Vr . The concatenation H(V) of all marginal histograms is used as a non-parametric approximation to the distribution of the primitive objects in the compound structure. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 12 / 78
  • 13. Compound Structure Model Spatial Arrangement Model Figure: Example histograms for the building layers of four different types of compound structures. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 13 / 78
  • 14. Compound Structure Model Probabilistic Region Processes Each primitive object vi (i.e., the ellipse parameters) is considered a vector-valued random variable. A compound structure is represented by a set of random variables that leads to a region process. The region process is governed by the Gibbs distribution p(V|β) = 1 Zv exp βT H(V) (1) where β is the parameter vector controlling each histogram bin, and Zv is the partition function. A region process is equivalent to a Markov random field (MRF). H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 14 / 78
  • 15. Learning Maximum Likelihood Estimation Suppose that we observe a set of i.i.d. region processes V = {V1, . . . , VM}. We can estimate a compound structure model via maximum likelihood estimation (MLE) of β by maximizing (β|V) = M m=1 log p(Vm|β). (2) The gradient of the log-likelihood is given by d (β|V) dβ = Ep[H(V)] − 1 M N m=1 H(Vm). (3) We use the stochastic gradient ascent algorithm where the expectation Ep[H(V)] is approximated by a finite sum of histograms of samples V(s), s = 1, . . . , S. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 15 / 78
  • 16. Learning Maximum Likelihood Estimation Figure: An example iteration for updating β corresponding to the relative orientation histogram bins. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 16 / 78
  • 17. Learning Sampling Region Processes (a) (b) t = 0 (c) t = 50 (d) t = 200 (e) t = 600 (f) t = 1000 Figure: Illustration of the Gibbs sampler for two primitive layers. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 17 / 78
  • 18. Inference and Region Selection Hierarchical Region Extraction Given a compound structure model with learned parameter vector β, we would like to automatically detect all of its instances in an input image. The detection problem is posed as the selection of multiple subgroups of candidate regions coming from multiple hierarchical segmentations. Figure: Hierarchical segmentation trees for two primitive layers. Each selected group of regions constitutes an instance of the example compound structure in the large image. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 18 / 78
  • 19. Inference and Region Selection Hierarchical Region Extraction Given a compound structure model with learned parameter vector β, we would like to automatically detect all of its instances in an input image. The detection problem is posed as the selection of multiple subgroups of candidate regions coming from multiple hierarchical segmentations. Figure: Hierarchical segmentation trees for two primitive layers. Each selected group of regions constitutes an instance of the example compound structure in the large image. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 18 / 78
  • 20. Inference and Region Selection Hierarchical Region Extraction The first step is the identification of candidate regions for each layer Vr by using a hierarchical segmentation algorithm. The next step is to connect the potentially related vertices at all levels to represent the neighbor relationships. Within-level edges (⊆ Er1r2 , r1 = r2): Voronoi tessellations. Between-level edges (⊆ Er1r2 , r1 = r2): Ancestor-descendant relations. Between-layer edges (⊆ Er1r2 , r1 = r2): Proximity-based neighbors. Figure: Hierarchical segmentation trees for two primitive layers. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 19 / 78
  • 21. Inference and Region Selection Hierarchical Region Extraction The first step is the identification of candidate regions for each layer Vr by using a hierarchical segmentation algorithm. The next step is to connect the potentially related vertices at all levels to represent the neighbor relationships. Within-level edges (⊆ Er1r2 , r1 = r2): Voronoi tessellations. Between-level edges (⊆ Er1r2 , r1 = r2): Ancestor-descendant relations. Between-layer edges (⊆ Er1r2 , r1 = r2): Proximity-based neighbors. Figure: Hierarchical segmentation trees for two primitive layers. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 19 / 78
  • 22. Inference and Region Selection Hierarchical Region Extraction Figure: Graph construction for two primitive layers (i.e., building and pool). The hierarchical candidate regions at three and two levels for these layers are shown in red and light blue, respectively. The edges that represent parent-child relationship for both layers are shown. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 20 / 78
  • 23. Inference and Region Selection Hierarchical Region Extraction Figure: Graph construction for two primitive layers (i.e., building and pool). The edges that represent the within- and between-level neighbor relationship within the same layer are shown. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 21 / 78
  • 24. Inference and Region Selection Hierarchical Region Extraction Figure: Graph construction for two primitive layers (i.e., building and pool). The edges that represent the within- and between-level neighbor relationship between the layers are shown. For better visualization of edges, only 20 percent of all between-layer edges are shown. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 22 / 78
  • 25. Inference and Region Selection Inference without Constraints Bayesian Formulation Given a graph G = (V, E), the problem can be formulated as the selection of a subset V∗ among all regions V as V∗ = arg max V ⊆V p(V |I) = arg max V ⊆V p(I|V )p(V ) (4) where p(I|V ) is the observed spectral data likelihood for the compound structure in the image, and p(V ) acts as the spatial prior according to the learned appearance and arrangement model. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 23 / 78
  • 26. Inference and Region Selection Inference without Constraints CRF Formulation We formulate the selection problem in (4) using a conditional random field (CRF). Let X = {x1, . . . , xM} where xi ∈ {0, 1}, i = 1, . . . , M, be the set of indicator variables associated with the vertices V of G so that xi = 1 implies region vi being selected. Our CRF formulation defines a posterior distribution as p(X|I, V) ∝ p(I|X, V)p(X, V) = 1 Zx vi ∈V exp ψc i + ψs i xi (vi ,vj )∈E exp ψa ij xixj . (5) H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 24 / 78
  • 27. Inference and Region Selection Inference without Constraints CRF Formulation The vertex bias terms ψc and ψs representing color and shape, respectively, and edge weights ψa representing arrangement are defined as ψc i = −1 2 (yi − µr )T (Σr )−1 (yi − µr ), ∀vi ∈ Vr , r = 1, . . . , R (6) ψs i = 9 k=6 βr k,Ir k φk (vi ) , ∀vi ∈ Vr , r = 1, . . . , R (7) ψa ij = 5 k=1 βr1r2 k,I r1r2 k φk (vi ,vj ) , ∀(vi, vj) ∈ E, r1, r2 = 1, . . . (8) H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 25 / 78
  • 28. Inference and Region Selection Inference without Constraints CRF Inference Selecting V∗ in (4) is equivalent to estimating the joint MAP labels given by X∗ = arg max X p(X|I, V). (9) Exact inference of the CRF formulation is intractable in general graphs. An approximate solution can be obtained by a Markov chain Monte Carlo sampler. We developed a sampling algorithm that samples the labels of many variables at once. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 26 / 78
  • 29. Inference and Region Selection Inference without Constraints CRF Inference Figure: Illustration of the primitive sampling procedure. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 27 / 78
  • 30. Inference with Constraints Our objective is to obtain the maximum probability estimates of the indicator variables, xi, i = 1, . . . , N satisfying convex inequality and equality constraints. We reformulate the problem as quadratic programming under convex constraints. The problem in Equation (4) can be rewritten as V∗ = arg max V ⊆V V ⊆Ω p(V |I) = arg max V ⊆V V ⊆Ω p(I|V )p(V ). (10) where Ω ∈ RN is a nonempty polyhedral convex set determined by a set of constraints. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 28 / 78
  • 31. Inference with Constraints Quadratic Programming Formulation For a V ⊆ V, log p(V |β∗ ) can be written as follows log p(V |β) = 5 k=1 R r1=1 R r2=1 (vi ,vj )∈Er1r2 βr1r2 k,I r1r2 k φk (vi ,vj ) xixj + 9 k=6 R r=1 vi ∈Vr βr k,Ir k φk (vi ) xi − log ZX . (11) where ZX is the partition function. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 29 / 78
  • 32. Inference with Constraints Quadratic Programming Formulation Let W = 5 k=1 R r1=1 R r2=1 Wr1r2 k where each Wr1r2 k is an N × N affinity matrix. Each element of this matrix is calculated as Wr1r2 k (i, j) = −βr1r2 k,I r1r2 k φk (vi ,vj ) . Also, let q = 9 k=6 R r=1 qr k where each qr k is an N × 1 potential vector. Each element of this vector is calculated as qr k (i) = βr k,Ir k φk (vi ) . The problem can be formulated as minimize x − log p(V |β) = 1 2 XT WX + qT X + log ZX subject to X ∈ Ω, X ∈ {0, 1}. (12) H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 30 / 78
  • 33. Inference with Constraints DC Programming Inference The problem can be formulated as minimize x − log p(V |β) = 1 2 XT WX + qT X + log ZX subject to X ∈ Ω, X ∈ {0, 1}. (13) First, a linear programming relaxation is applied to the 0 − 1 integer program so that 0 ≤ x ≤ 1. Since W is not assumed positive semidefinite, the resulting linearly constrained quadratic problem is not convex. The objective function can be reformulated as a difference of two convex functions. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 31 / 78
  • 34. Inference with Constraints Difference of Convex Programming Formulation A Difference of Convex (DC) problem is defined as P = min f(X) = g(X) − h(X) : X ∈ RN (14) where g : RN → R and h : RN → R are convex functions. Consider the dual program D = min f∗ (Y) = h∗ (Y) − g∗ (Y) : Y ∈ RN (15) where g∗ is the conjugate function of g. An iterative primal-dual algorithm constructs two alternating sequences {X(t) } and {Y(t) } such that g(X(t)) − h(X(t)) and g∗(Y(t)) − h∗(Y(t)) are decreasing, converging to the optimal solutions, X∗ and Y∗, to the primal and dual problems, respectively. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 32 / 78
  • 35. Inference with Constraints DC Programming Inference Let W = QΛQT be the eigenvalue decomposition of W, and Λ+ = diag(Λ+ 1 , . . . , Λ+ N ) (respectively, Λ− = diag(Λ− 1 , . . . , Λ− N )) be the positive semidefinite diagonal matrix (respectively, negative semidefinite diagonal matrix) of Λ. We rewrite the nonconvex symmetric quadratic objective function as 1 2 XT WX + qT X = g(X) − h(X) g(X) = 1 2 XT W+ X + qT X + χΩ(X) h(X) = − 1 2 XT W− X (16) where W+ = QΛ+ QT , W− = QΛ− QT , and χΩ(X) is the space enclosed by the constraints X ∈ {0, 1} and Ω. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 33 / 78
  • 36. Inference with Constraints Experiments We present detailed results of four different kinds of experiments: 1 Using a single layer without imposing any constraint. 2 Using multiple layers without imposing any constraint. 3 Using a single layer by imposing constraints. 4 Using multiple layers by imposing constraints. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 34 / 78
  • 37. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Figure: 2500 × 4000 pixels Ankara data set. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 35 / 78
  • 38. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Table: Detection scenarios for the experiments. Example primitives used for learning the compound structure model for each scenario are shown in a different color. The number of polygons and buildings in the validation data are also given. Scenario 1 2 3 4 5 6 Example primitives # polygons 162 98 48 195 60 16 # buildings 1519 870 1117 1796 771 219 H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 36 / 78
  • 39. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Figure: Candidate regions hierarchy. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 37 / 78
  • 40. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Figure: Marginal probabilities for the first scenario. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 38 / 78
  • 41. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Figure: Marginal probabilities for the second scenario. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 39 / 78
  • 42. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Figure: Marginal probabilities for the third scenario. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 40 / 78
  • 43. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Figure: Marginal probabilities for the fourth scenario. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 41 / 78
  • 44. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Figure: Marginal probabilities for the fifth scenario. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 42 / 78
  • 45. Experiments Unconstrained Single-layer Experiments Results-Urban Structures * Figure: Marginal probabilities for the sixth scenario. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 43 / 78
  • 46. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Figure: Zoomed detection examples. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 44 / 78
  • 47. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Figure: Zoomed detection examples. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 45 / 78
  • 48. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Figure: Zoomed detection examples. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 46 / 78
  • 49. Experiments Unconstrained Single-layer Experiments Results-Urban Structures Figure: Zoomed detection examples. Each row corresponds to a particular scenario. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 47 / 78
  • 50. Experiments Unconstrained Single-layer Experiments Results-Orchards Figure: 3000 × 8000 pixels Kusadasi data set. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 48 / 78
  • 51. Experiments Unconstrained Single-layer Experiments Results-Orchards Figure: (Up) Candidate regions. (Down) Selected regions. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 49 / 78
  • 52. Experiments Unconstrained Single-layer Experiments Results-Orchards Figure: Example results for the detection of orchards in the subimage on the left column. The right column shows the corresponding marginal probabilities of the selected regions (the copper colormap) as well as the discarded input candidate regions (white). H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 50 / 78
  • 53. Experiments Unconstrained Single-layer Experiments Results-Orchards Figure: Example results for the detection of orchards in the subimage on the left column. The right column shows the corresponding marginal probabilities of the selected regions (the copper colormap) as well as the discarded input candidate regions (white). H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 51 / 78
  • 54. Experiments Unconstrained Single-layer Experiments Results-Orchards Figure: Example results for the detection of orchards. The left column shows the marginal probabilities at the end of selection. The right column shows the thresholded detections overlayed as red. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 52 / 78
  • 55. Experiments Unconstrained Single-layer Experiments Results-Refugee Camps Figure: 1102 × 971 pixels Darfur data set. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 53 / 78
  • 56. Experiments Unconstrained Single-layer Experiments Results-Refugee Camps Figure: Example results for the detection of refugee camps as rural structures. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 54 / 78
  • 57. Experiments Unconstrained Multi-layer Experiments Results-Housing Estates Figure: 3000 × 8000 pixels Kusadasi data set. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 55 / 78
  • 58. Experiments Unconstrained Multi-layer Experiments Results-Housing Estates Figure: (Up) Examples of local details of red building rooftops. (Down) An example hierarchy. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 56 / 78
  • 59. Experiments Unconstrained Multi-layer Experiments Results-Housing Estates The selection algorithm that used only the building layer could not detect several housing estates. The idea was to add a pool layer that can provide additional cues for finding the missed buildings. The initial model was extended by learning the arrangements of buildings with respect to pools as well. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 57 / 78
  • 60. Experiments Unconstrained Multi-layer Experiments Results-Housing Estates Figure: (Up) Candidate regions. (Down) Selected regions from the building and pool layers.H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 58 / 78
  • 61. Experiments Unconstrained Multi-layer Experiments Results-Housing Estates Table: The number of candidate and detected regions for single and multi-layer selection scenarios. Single-layer Multi-layer Candidates Detected Candidates Detected Building 67,983 11,173 67,983 11,871 Pool - - 16,276 436 Total 67,983 11,173 84,259 12,307 H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 59 / 78
  • 62. Experiments Unconstrained Multi-layer Experiments Results-Housing Estates Figure: Samples obtained by the selection procedure ran on single and multiple layers. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 60 / 78
  • 63. Experiments Unconstrained Multi-layer Experiments Results-Housing Estates (a) (b) Figure: The selected regions using (a) only the building layer. (b) building and pool layers. Newly detected housing estates that was missed with single layer selection is enclosed by a red convex hull. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 61 / 78
  • 64. Experiments Unconstrained Multi-layer Experiments Results-Housing Estates (a) (b) Figure: The selected regions using (a) only the building layer. (b) building and pool layers. Newly detected housing estates that was missed with single layer selection is enclosed by a red convex hull. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 62 / 78
  • 65. Experiments Unconstrained Multi-layer Experiments Results-Housing Estates Figure: Ground view of a missed housing estate with single layer selection. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 63 / 78
  • 66. Experiments Constrained Single-layer Experiments Figure: Zoomed detection examples. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 64 / 78
  • 67. Experiments Problem Definition Unconstrained selection involved overlapping regions at different levels of the hierarchy. To overcome this problem, we require at most one region should be selected per path where a path corresponds to the set of vertices from a leaf to the root. Formally, we select an optimal subset V∗ ⊆ V such that ∀a, b ∈ V∗ , a ∈ descendant(b) and b ∈ descendant(a). H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 65 / 78
  • 68. Experiments Constrained Single-layer Experiments Let A be a |P| × |V| matrix. P denotes all the paths from the leaves to the roots of the input hierarchical forest. A(i, j) = 1 implies vi ∈ pj ∈ P. The problem can be reformulated as minimize x 1 2 XT WX + qT X + log ZX subject to AX ≤ 1, 0 ≤ X ≤ 1. (17) The resulting problem is solved by the DC inference algorithm. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 66 / 78
  • 69. Experiments Results-Urban Structures Table: The number of selected regions for unconstrained and constrained selection scenarios. # cand.s 70,644 70,644 70,644 70,644 70,644 22,195 Uncnstr. 3191 1828 3819 3201 2027 1612 Cnstr. 1485 856 2562 1740 811 263 H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 67 / 78
  • 70. Experiments Results-Urban Structures (a) (b) Figure: Zoomed detection examples. (a) shows the RGB image for a 300 × 300 sub-scene. (b) shows the hierarchy of candidate regions (two-level hierarchy from bottom to top). H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 68 / 78
  • 71. Experiments Results-Urban Structures (a) (b) Figure: Zoomed detection examples. (a) shows the RGB image for a 300 × 300 sub-scene. (b) shows the hierarchy of candidate regions (six-level hierarchy from bottom to top). H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 69 / 78
  • 72. Experiments Constrained Multi-layer Experiments The last set of experiments uses two primitive layers and enforces geometrical constraints between them. We search for nearby alike buildings and green areas where each building group must have a green area in the middle. We strictly require that the distance between the centroid of the centroids of a selected group of similar buildings and the centroid of a selected large green area cannot exceed a distance threshold δ . H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 70 / 78
  • 73. Experiments Constrained Multi-layer Experiments Let V1 and V2 represent the building and green area layers. The desired set of regions can be selected by minimize x 1 2 XT WX + qT X + log ZX subject to AX ≤ 1, 1 k1 vi ∈V1 sh i xi − 1 k2 vj ∈V2 sh j xj ≤ δ 1 k1 vi ∈V1 sw i xi − 1 k2 vj ∈V2 sw j xj ≤ δ vi ∈V1 xi = k1 vj ∈V2 xj = k2 0 ≤ X ≤ 1. (18) where (sh i , sw i ) is the centroid of region vi, k1 and k2 denote the number of buildings and green areas to be selected. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 71 / 78
  • 74. Experiments Constrained Multi-layer Experiments Results-Buildings & Green Areas Figure: 2500 × 4000 pixels Ankara data set. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 72 / 78
  • 75. Experiments Constrained Multi-layer Experiments Results-Buildings & Green Areas Figure: Selected regions for the green areas surrounded by buildings. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 73 / 78
  • 76. Experiments Constrained Multi-layer Experiments Results-Buildings & Green Areas (a) RGB (b) Building candidates (c) Green candi- dates (d) Selection (e) Overlay Figure: A zoomed detection example for k1 = 4, k2 = 1. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 74 / 78
  • 77. Experiments Constrained Multi-layer Experiments Results-Buildings & Green Areas (a) RGB (b) Building candidates (c) Green candi- dates (d) Selection (e) Overlay Figure: A zoomed detection example for k1 = 6, k2 = 1. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 75 / 78
  • 78. Experiments Constrained Multi-layer Experiments Results-Buildings & Green Areas (a) k1 = 4, fVal = −2.95 (b) k1 = 8, fVal = −3.23 (c) k1 = 6, fVal = −3.16 (d) k1 = 4, fVal = −2.97 Figure: Zoomed detection examples for different values of k1 = 4, 6, 8. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 76 / 78
  • 79. Experiments Summary & Conclusions We described a generic method for the modeling and detection of compound structures that consisted of arrangements of mostly unknown number of primitives. The modeling process built an MRF-based contextual model for the compound structure of interest. The detection task involved a combinatorial selection problem where multiple subsets of candidate regions from multiple hierarchical segmentations were selected. We also handled hard constraints on the candidate regions. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 77 / 78
  • 80. Experiments Summary & Conclusions Experiments using urban, industrial, agricultural and rural structures showed that the proposed method can provide good localization of instances of compound structures. The multi-layered experiments showed that selection of some objects required the selection of objects in other layers that had spatial relation with them. One of the most important bottlenecks in terms of accuracy was the errors in the input hierarchical segmentations. Future work includes Using the detection results for adjusting wrong segmentation results. Inferring the primitive objects inside a compound structure. H. G. Akc¸ay Compound Structure Detection 21 Sept. 2016 78 / 78