1. Introduction
Development
Experiments
Conclusions
Large Scale Semisupervised Image Segmentation
With Active Queries
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı
Image Processing Laboratory
University of Valencia, Spain
IGARSS 2011, Vancouver, Canada
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

2. Introduction
Development Introduction
Experiments Motivation
Conclusions
Introduction
Outline:
Image segmentation using a hierarchical description of the image
Hierarchical description based on clustering
Use active learning procedures to
Converge faster to an optimal solution
... and improve segmentation results
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

3. Introduction
Development Introduction
Experiments Motivation
Conclusions
Cluster based segmentation
Problems
1 Find right number of clusters
2 Find correct cluster labels
Undersegmentation Good level of segmentation Oversegmentation
Wrong labeling of big clusters Correct labeling Wrong labeling of small clusters
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

4. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Active Learning Segmentation
Proposed methodology components:
1 A hierarchical description of the data
Bottom up: linkage (slow, unfeasible for large images)
Top down: k-means (fast, proposed implementation)
2 Adaptation rule
Prune the description above to adapt it to a description according to
the objects and classes defined by the user
3 Active selection
The algorithm selects the samples to label that will improve results
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

5. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adapting the hierarchical description
Nodes level
Hierarchical Description Segmentation
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

6. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adaptation: overall procedure
1 Obtain a hierarchical
description
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

7. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adaptation: overall procedure
1 Obtain a hierarchical
description
2 Descend through the tree
and ask the user for sample
labels
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

8. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adaptation: overall procedure
1 Obtain a hierarchical
description
2 Descend through the tree
and ask the user for sample
labels
3 Ascend and update node
labels
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

9. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adaptation rule: labeling
Get labels and estimate
pv ,l ∼ # labels l on node v
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

10. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adaptation rule: labeling
Get labels and estimate
pv ,l ∼ # labels l on node v
LB UB
Conf. interval [pv ,ω , pv ,ω ]
LB
pv ,ω = max(pv ,ω − ∆v ,ω , 0)
UB
pv ,ω = min(pv ,ω + ∆v ,ω , 1)
∆v ,ω ∝ node size and
number of labeled samples
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

11. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adaptation rule: labeling
Get labels and estimate
pv ,l ∼ # labels l on node v
LB UB
Conf. interval [pv ,ω , pv ,ω ]
LB
pv ,ω = max(pv ,ω − ∆v ,ω , 0)
UB
pv ,ω = min(pv ,ω + ∆v ,ω , 1)
∆v ,ω ∝ node size and
number of labeled samples
LB UB
pv ,l > 2pv ,l − 1 ∀l = l
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

12. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adaptation rule: labeling
Get labels and estimate
pv ,l ∼ # labels l on node v
LB UB
Conf. interval [pv ,ω , pv ,ω ]
LB
pv ,ω = max(pv ,ω − ∆v ,ω , 0)
UB
pv ,ω = min(pv ,ω + ∆v ,ω , 1)
∆v ,ω ∝ node size and
number of labeled samples
LB UB
pv ,l > 2pv ,l − 1 ∀l = l
Compute all admissible
labels and take the winner
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

13. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adaptation rule: error estimation
Update tree:
Error estimation of labeling a
node as ω:
1 − pv ,ω if (v , ω) admissible
˜v ,ω =
1 otherwise
Divide if
˜v ,ω > (˜vl ,ωl + ˜vr ,ωr )
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

14. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adaptation rule: error estimation
Update tree:
Error estimation of labeling a
node as ω:
1 − pv ,ω if (v , ω) admissible
˜v ,ω =
1 otherwise
Divide if
˜v ,ω > (˜vl ,ωl + ˜vr ,ωr )
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

15. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adaptation rule: error estimation
Update tree:
Error estimation of labeling a
node as ω:
1 − pv ,ω if (v , ω) admissible
˜v ,ω =
1 otherwise
Divide if
˜v ,ω > (˜vl ,ωl + ˜vr ,ωr )
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

16. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Adaptation rule: error estimation
Update tree:
Error estimation of labeling a
node as ω:
1 − pv ,ω if (v , ω) admissible
˜v ,ω =
1 otherwise
Good Pruning !
Divide if
˜v ,ω > (˜vl ,ωl + ˜vr ,ωr )
At the end each node has
An estimated error (˜v ,ω )
LB
A confidence (pv ,l )
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

17. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Active learning to select nodes and subnodes
Active Learning is about obtaining better results labeling less, but better.
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

18. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Active learning to select nodes and subnodes
Active Learning is about obtaining better results labeling less, but better.
Node selection strategies
s0 Proportional to node size (∼ random sampling): nv
LB
s1 Proportional to node size and uncertainty: nv · pv
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

19. Introduction
Active Learning Segmentation
Development
Adapting the hierarchical description
Experiments
Active learning node selection
Conclusions
Active learning to select nodes and subnodes
Active Learning is about obtaining better results labeling less, but better.
Node selection strategies
s0 Proportional to node size (∼ random sampling): nv
LB
s1 Proportional to node size and uncertainty: nv · pv
Subnode selection strategies (left of right node’s child)
d0 Proportional to subnode size: nv
LB
d1 Proportional to subnode uncertainty: pv
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

20. Introduction
Data description
Development
Results
Experiments
Visual inspection
Conclusions
Experiments
145 × 145 AVIRIS image
Indian Pines area, Indiana
Spatial resolution: 30 m
16 crop classes
200 spectral bands (0.4 -
2.5 µm)
All the available 10366 pixels
are considered
Spectral + spatial + PCA
Clustering: hierarchical
k-means (top-down)
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

21. Introduction
Data description
Development
Results
Experiments
Visual inspection
Conclusions
AVIRIS results
Mean results over 10 realizations
100
Random
90 Active
80
70
60
Error (%)
50
40
30
20
10
0
0 200 400 600 800 1000 1200 1400
Num. sample
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

22. Introduction
Data description
Development
Results
Experiments
Visual inspection
Conclusions
Visual inspection (50 labeled samples)
Ground truth Classification Confidence 10 Clusters
Random
Active
10 clusters
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

23. Introduction
Data description
Development
Results
Experiments
Visual inspection
Conclusions
Visual inspection (100 labeled samples)
Ground truth Classification Confidence 15 Clusters
Random
Active
21 clusters
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

24. Introduction
Data description
Development
Results
Experiments
Visual inspection
Conclusions
Visual inspection (200 labeled samples)
Ground truth Classification Confidence 34 Clusters
Random
Active
46 clusters
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

25. Introduction
Data description
Development
Results
Experiments
Visual inspection
Conclusions
Visual inspection (400 labeled samples)
Ground truth Classification Confidence 55 Clusters
Random
Active
82 clusters
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

26. Introduction
Development
Conclusions
Experiments
Conclusions
Conclusions
Structure-based AL exploits cluster structure of data
It discovers the structure representing the user’s desired classes
It does not need a starting training set or fixing the number of
classes
It is fast (no model is required)
Classification and confidence maps are obtained
With a bad clustering, slower convergence
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries

27. Introduction
Development
Conclusions
Experiments
Conclusions
Large Scale Semisupervised Image Segmentation
With Active Queries
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı
Image Processing Laboratory
University of Valencia, Spain
IGARSS 2011, Vancouver, Canada
Devis Tuia, Jordi Mu˜oz-Mar´ and Gustavo Camps-Valls
n ı Large Scale Semisupervised Image Segmentation With Active Queries