This paper proposes a new method for classifying multitemporal high-resolution images using Markov random fields and multiscale segmentation. The method fuses spatial, temporal, and multiscale information by generating segmentation maps at different scales and time points, and using these in a novel region-based multitemporal Markov random field model. Experimental results on QuickBird and SPOT-5 datasets show the method produces accurate classification maps and outperforms previous techniques that do not fully leverage multiscale and contextual information.

1. IGARSS-2011
Vancouver, Canada, July 24-29, 2011
Multitemporal Region-Based Classification
of High-Resolution Images by Markov
Random Fields and Multiscale
Segmentation
Gabriele Moser
Sebastiano B. Serpico
University of Genoa Department of Biophysical
and Electronic Engineering

2. 2
Outline
• Introduction
– Multitemporal high-resolution image classification
• The proposed method
– Overall architecture
– Multitemporal multiscale region-based Markov random field
– Parameter estimation and energy minimization
• Experimental results
– Data sets and experimental set-up
– Classification maps
– Classification accuracies and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

3. 3
Outline
• Introduction
– Multitemporal high-resolution image classification
• The proposed method
– Overall architecture
– Multitemporal multiscale region-based Markov random field
– Parameter estimation and energy minimization
• Experimental results
– Data sets and experimental set-up
– Classification maps
– Classification accuracies and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

4. 4
Introduction
Peking, SPOT-5, 2003
SPOT-
• Multitemporal high-resolution (HR) optical image
classification:
– Important for environmental monitoring (e.g. in
disaster prevention and management), urban area
analysis, etc.
– 0.5 ÷ 10 m resolution by current (e.g., IKONOS,
QuickBird, WorldView-2, GeoEye-1, SPOT-5 HRG)
and forthcoming (e.g., Pleiades) missions.
Peking, SPOT-5, 2005
SPOT-
– Need for modeling the temporal and spatial-
geometrical information in multitemporal HR data.
• A supervised multitemporal multiscale region-
based HR image classifier is proposed, based on:
– Multiscale segmentation;
– Markov random field (MRF) models.
University of Genoa Department of Biophysical
and Electronic Engineering

5. 5
Outline
• Introduction
– Multitemporal high-resolution image classification
• The proposed method
– Overall architecture
– Multitemporal multiscale region-based Markov random field
– Parameter estimation and energy minimization
• Experimental results
– Data sets and experimental set-up
– Classification maps
– Classification accuracies and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

6. 6
The Proposed Method
• Motivation
– Key role of spatial-contextual and multiscale
information in HR image analysis.
– Need for modeling the temporal context
associated to multitemporal images.
• Key-ideas
– Extracting multiscale information at each acquisition
time by generating segmentation maps at different
scales.
– Fusing spatial, temporal, and multiscale information by
a novel region-based MRF model
– The model extends two previous models for single-time
multiscale and single-scale multitemporal classification,
respectively.
University of Genoa Department of Biophysical
and Electronic Engineering

7. 7
Architecture of the Proposed Method
Segmentation map S10
at the finest scale
HR Graph-based
image X0 segmentation
at time t0 Segmentation map S20
at intermediate scale
Classification
Training ··· map for time t0
map for
Segmentation map SQ0
time t0
at the coarsest scale
MRF-based multiscale
and multitemporal
fusion
Segmentation map S11
Training
at the finest scale
map for
time t1 Classification
Segmentation map S21 map for time t1
at intermediate scale
HR Graph-based
image X1 segmentation ···
at time t1 Segmentation map SQ1
at the coarsest scale
University of Genoa Department of Biophysical
and Electronic Engineering

8. 8
Outline
• Introduction
– Multitemporal high-resolution image classification
• The proposed method
– Overall architecture
– Multitemporal multiscale region-based Markov random field
– Parameter estimation and energy minimization
• Experimental results
– Data sets and experimental set-up
– Classification maps
– Classification accuracies and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

9. 9
Multitemporal Markov Random Fields
• MRF model for the spatio-temporal context
– Representation of the statistical interactions between the pixel
labels at each time tr (r = 0, 1) by using only local relationships
in both the spatial and temporal domains:
( ) (
P ℓ ir { ℓ jr } j ≠i ,{ ℓ k ,1−r } = P ℓ ir { ℓ jr } j ∼i , { ℓ k ,1−r } k ≃ i )
Conditioning only to the
labels of the (spatially or Time tr Time t1 – r
temporally) neighboring
pixels, according to a j
given neighborhood i i
system (here 3 × 3)
k
Spatial context of the i-th pixel Temporal context of the i-th pixel
in the image at each time tr in the image at the other time t1 – r
University of Genoa Department of Biophysical
and Electronic Engineering

10. 10
Energy Function
• MRF-based classification
– Minimization of a (posterior) energy function U(·), thanks to
the Hammersley-Clifford theorem.
– When using MRFs for data fusion, U(·) is a linear combination
of energy contributions, each related to an information source.
• Proposed region-based multitemporal MRF model
– Fusion of spatial, temporal, and multiscale information:
Transition
1
 Q
U ( L0 , L1 S0 ,S1 ) = − ∑  ∑∑ α qr ln P (siqr | ℓ ir ) + probability from
r =0  i q =1 each class at a
time to each
 class at the
Pixelwise probability mass + βr ∑ δ (ℓ ir , ℓ jr ) + γ r ∑ P (ℓ ir | ℓ k ,1−r ) other time
function (PMF) of the segment i∼j i ≃k 
labels in the segmentation map
at each scale and each date, Potts model for the
conditioned to each class spatial context
University of Genoa Department of Biophysical
and Electronic Engineering

11. 11
Segmentation and PMF Estimation
• Felzenszwalb & Huttenlocherm segmentation method [3]
– Graph-based region-growing method depending on a scale
parameter.
– Segmentations at different scales by varying this parameter.
• Class-conditional PMF estimation
– Extension of a previous method proposed for a single-time
region-based MRF.
– Relative-frequency estimate, based on preliminary
classification maps M0 and M1.
– M0 and M1 generated here by a classical MRF classifier with
k-NN estimates of the pixelwise class-conditional statistics.
University of Genoa Department of Biophysical
and Electronic Engineering

12. 12
Outline
• Introduction
– Multitemporal high-resolution image classification
• The proposed method
– Overall architecture
– Multitemporal multiscale region-based Markov random field
– Parameter estimation and energy minimization
• Experimental results
– Data sets and experimental set-up
– Classification maps
– Classification accuracies and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

13. 13
Parameter Estimation
and Energy Minimization
• Transition probabilities: expectation-maximization (EM)
– Converges (under mild assumptions) to ML estimates.
– Uses the aformentioned k-NN pixelwise estimates.
• Weight parameters (α, β, γ) in the MRF
– Extension of a recent method based on the Ho-Kashyap
algorithm.
• Energy minimization: iterated conditional mode (ICM)
– Initialized with the preliminary maps M0 and M1.
– Converges to a local energy minimum.
– Usually good tradeoff between accuracy and time.
University of Genoa Department of Biophysical
and Electronic Engineering

14. 14
Outline
• Introduction
– Multitemporal high-resolution image classification
• The proposed method
– Overall architecture
– Multitemporal multiscale region-based Markov random field
– Parameter estimation and energy minimization
• Experimental results
– Data sets and experimental set-up
– Classification maps
– Classification accuracies and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

15. 15
Data Sets and Experimental Set-up
• SPOT-5 HRG data set
2003 – Peking (China), 4 bands, 1500 ×
1160 pixels, 10-m resolution,
acquisitions in 2003 and 2005.
• QuickBird data set
– Phoenix (AZ), 4 bands, 300 × 300
pixels, 2.8-m resolution,
acquisitions in 2003 and 2005.
2005
• Experimental comparisons
– With a previous multitemporal
non-region-based single-scale
classifier based on MRFs [6].
2005 – With a previous multitemporal
non-contextual single-scale
Peking: false color RGB Phoenix: true color RGB classifier based on EM [8].
University of Genoa Department of Biophysical
and Electronic Engineering

16. 16
Outline
• Introduction
– Multitemporal high-resolution image classification
• The proposed method
– Overall architecture
– Multitemporal multiscale region-based Markov random field
– Parameter estimation and energy minimization
• Experimental results
– Data sets and experimental set-up
– Classification maps
– Classification accuracies and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

17. 17
“Phoenix”: Classification Maps
2003
2005
True color RGB EM method in [8] MRF method in [6] Proposed method urban
shrub-brush
herbaceous
barren
transitional
University of Genoa Department of Biophysical
and Electronic Engineering

18. 18
“Peking”: Classification Maps
False color RGB
urban
agricultural
2003 2005 rangeland
barren
water
Proposed method
University of Genoa Department of Biophysical
and Electronic Engineering

19. 19
Outline
• Introduction
– Multitemporal high-resolution image classification
• The proposed method
– Overall architecture
– Multitemporal multiscale region-based Markov random field
– Parameter estimation and energy minimization
• Experimental results
– Data sets and experimental set-up
– Classification maps
– Classification accuracies and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

20. 20
Classification Accuracies
– Application with up to 5 scales.
– Accurate classification maps with “Phoenix” (overall accuracy, OA ≈ 94 ÷
98%; average accuracy, AA ≈ 92 ÷ 96%).
– Quite accurate maps with “Peking” (OA ≈ 84%, AA ≈ 85%): strong spectral
overlapping between the classes (especially “urban vs. barren land”).
University of Genoa Department of Biophysical
and Electronic Engineering

21. 21
Experimental Comparisons
– More accurate results by the method in [8] (MRF fusion of spatio-temporal
context) than by the one in [6] (modeling only temporal correlation).
– More accurate results by the proposed method than by both previous
techniques. This suggests the effectivenes of the method in fusing spatial,
temporal, and multiscale information for region-based multitemporal
classification.
University of Genoa Department of Biophysical
and Electronic Engineering

22. 22
Outline
• Introduction
– Multitemporal high-resolution image classification
• The proposed method
– Overall architecture
– Multitemporal multiscale region-based Markov random field
– Parameter estimation and energy minimization
• Experimental results
– With QuickBird images
– With SPOT-5 HRG images
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

23. 23
Conclusion
• A novel multitemporal classifier has been proposed for
HR images, based on a region-based multiscale MRF.
– Capability to fuse spatio-temporal context and multiscale
information associated to a multitemporal HR data set.
– Accurate results with different sensors (QuickBird, SPOT-5)
and resolutions (2.8 m, 10 m).
– Accuracy improvements, compared to previous multitemporal
methods based on temporal or spatio-temporal models.
– Confirmation of the relevance of multiscale information in HR
image analysis.
• Possible future generalizations
– Integrating edge and/or texture information.
– Approaching global energy minimization (e.g., graph-cuts).
– Comparisons with other techniques for multitemporal VHR
classification
University of Genoa Department of Biophysical
and Electronic Engineering

24. 24
References
1. S. Li, Markov random field modeling in image analysis, Springer, 2009
2. A. H. S. Solberg, T. Taxt, and A. K. Jain, “A Markov random field model for classification of multisource
satellite imagery,” IEEE Trans. Geosci. Remote Sensing, vol. 34, no. 1, pp. 100–113, 1996
3. P. F. Felzenszwalb and D. Huttenlocherm, “Efficient graph-based image segmentation,” Int. J. Comp. Vis.,
vol. 59, pp. 167–181, 2004
4. F. Causa, G.Moser, and S. B. Serpico, “A novel MRF model for the detection of urban areas in optical high
spatial resolution images,” Rivista Italiana di Telerilevamento, vol. 38, 2007
5. G. Moser and S. B. Serpico, “Classification of high-resolution images based on MRF fusion and multiscale
segmentation,” in Proc. of IGARSS-2008, Boston, USA, 2008, vol. II, pp. 277–280
6. F. Melgani and S. B. Serpico, “A Markov random field approach to spatio-temporal contextual image
classification,” IEEE Trans. Geosci. Remote Sensing, vol. 41, no. 11, pp. 2478–2487, 2003
7. S. B. Serpico and G. Moser, “Weight parameter optimization by the Ho-Kashyap algorithm in MRF models
for supervised image classification,” IEEE Trans. Geosci. Remote Sensing, vol. 44, pp. 3695–3705, 2006
8. L. Bruzzone and S. B. Serpico, “An iterative technique for the detection of land-cover transitions in
multitemporal remote-sensing images,” IEEE Trans. Geosci. Remote Sensing, vol. 36, pp. 858–867, 1997
9. R. A. Redner and H. F. Walker, “Mixture densities, maximum likelihood, and the EM algorithm,” SIAM
Review, vol. 26, no. 2, pp. 195– 239, 1984
10. F. Bovolo, “A multilevel parcel-based approach to change detection in very high resolution multitemporal
images,” IEEE Geosci. Remote Sensing Lett., vol. 6, no. 1, pp. 33–37, 2009
University of Genoa Department of Biophysical
and Electronic Engineering