1. INNOVATIVE METHODS FOR THE RECONSTRUCTION
OF NEW GENERATION SATELLITE REMOTE SENSING
IMAGES
November 29th, 2012
PhD Student:
Luca Lorenzi
lorenzi@disi.unitn.it
Ph.D. thesis defense
Advisors:
Farid Melgani
Grégoire Mercier
2. Introduction – General Problem
Missing data in VHR optical image;
Mainly due to acquisition conditions, e.g., the presence of:
clouds: partially or completely missing data;
shadows: partially missing data.
2
MODIS: Black see
GeoEye-1: Doha Stadium, Qatar
3. Introduction – General Solution
A common solution approach:
1. Pre-process the image (co-registration, calibration);
2. Detect the location of the contaminated regions;
3. Attempt to restore the missing areas.
3
Missing Area
Detection
Missing Area
Reconstruction
Original
Image
Image Pre-
Processing
Restored
Image
4. Objective
Propose new methodologies for the reconstruction of
missing areas for new generation satellite remote
sensing images.
Missing areas due to the presence of:
Clouds
Shadows
4
5. Cloud-Contaminated Images
Contribution 1: different solutions based on the inpainting approach.
Three strategies:
local image properties;
isometric transformations;
multiresolution processing scheme.
Contribution 2: to improve the reconstruction process by integrating both
radiometric and spatial information, through a specific kernel.
Contribution 3: new methods based on the Compressive Sensing theory.
Three strategies:
Orthogonal Matching Pursuit (OMP);
Basis Pursuit (BP);
An alternative solution based on Genetic Algorithms (GAs).
5
6. Shadow-Contaminated Images
Contribution 4: a novel approach to solve both
problems of detection and reconstruction.
Shadow detection is performed through a hierarchical
supervised classification scheme, while the proposed
reconstruction relies on a linear prediction function, which
exploits information returned by the classification.
Contribution 5: to try to answer to the following
question: Is it possible to know a priori if a shadow
area can be well recovered?
Eight different criteria.
A fuzzy logic combination is explored.
6
8. MISSING AREA RECONSTRUCTION IN
MULTISPECTRAL IMAGES UNDER A COMPRESSIVE
SENSING PERSPECTIVE
PUBLISHED IN THE IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING,
VOL. 51, IN PRESS, 2013
L. Lorenzi, F. Melgani, and G. Mercier
9. Problem formulation
In I(1), any pixel can be expressed as:
We have to evaluate:
From I(1) :
In I(2) :
9
2,1,)()()(
iI iii
ΦΩ
)1()1(
Ωx
)1()1()1(
,, ΩαΦ lkx
)1()1(
, xf Φα
αΦ )2()2(
ˆx
f ?
source area
missing area
10. Compressive sensing (CS)
Compressive sensing theory [1]
Idea: exploit redundancy in signals
Signals like images are sparse → many coefficients close to zero
To enforce the sparsity constraint, CS finds a vector which minimizes:
D is a dictionary with a predefined number of atoms;
x is the original pixel, expressed by a linear combination of atoms;
Eq. (1) represents a NP-hard problem → computationally infeasible to
solve.
Candès and Tao [2], reduce the Eq. (1) in a relatively easy linear
programming solution:
under some reasonable assumptions:
10
αα Dxsubject tomin 0
0:#0
ii α
(1)
[1] D. L. Donoho, “Compressed Sensing”, IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289-1306, Apr 2006.
[2] E. J. Candès and T. Tao, “Decoding by Linear Programming”, IEEE Trans. Inform. Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005.
01
minmin
11. Orthogonal Matching Pursuit (OMP)
12
[3] Y. C. Pati, R. Rezaiifar and P. S. Krishnaprasad, “Orthogonal Matching Pursuit: Recursive Function Approximation with Applications to
Wavelet Decompositions”, in Proc. 27th Asilomar Conf. on Sig., Sys. and Comp., Nov. 1-3, 1993.
Orthogonal Matching Pursuit (OMP) [3] finds the atoms
which has the highest correlation with the signal:
where dictionary D is a collection of atom vectors and R(m) is a
residual.
It updates the coefficients of the selected atoms at
each iteration (adopting a least-squares step), so that
the resulting residual vector R is orthogonal to the
subspace spanned by the selected atoms.
)(
1
m
m
i
dd
Dd
dd Rx ii
Ddd
12. Orthogonal Matching Pursuit (OMP)
13
OMP pseudo code
i=0: x(0)=0 , R(0)=x and D(c0)={∅}
i=k:
i=m: ,)(
1
)( m
m
i
dd
m
Rx ii
)(m
RR
Step 1: find which ;
→ add to the set of selected variables;
→ update .
Step 2: let denote the projection onto the linear
space spanned by the elements of
→ update
Step 3: compute s.t.
Step 4: if ||R(k)||<th, stop, else set k=k+1 and return to Step 1
)1(
max kT
j
j
R
kj
kkk jcc 1
T
kk
T
kkk ccccP
1
icD
kj
xPIR k
k
)(
)(k
x
i
i
ci
d
k
i
k
x )()(
1.
2.
3.
kk
k
vAb 1)(
)1,...,1()()()1()(
kib k
i
k
k
k
i
k
i
k
i
jj
k
i
j
Tk
k
k
ki
k
b
R
1
)(
)(
)(
,1
13. Basis Pursuit (BP)
To solve Eq. (1), we may adopt the Basis Pursuit
principle [4]: convexification from L0 to L1;
Thanks to that, it becomes a support minimization problem;
Eq. (2) can be reformulated as a linear programming
(LP) problem, and solved using the Simplex methods.
Given that, it is possible to rewrite L1 norm in Eq. (2) as:
where
If we substitute it in Eq. (2), it allows to perform a linear
minimization problem.
14
[4] S. S. Chen, D. L. Donoho and M. A. Saunders, “Atomic Decomposition by Basis Pursuit”, SIAM J. on Sci. Comp., vol. 20, pp. 33-61, 1999.
αα Dxsubject tomin 1
(2)
i iii i vu1
α
00,
00,
iiii
iiii
ifuv
ifvu
14. Genetic Algorithm (GA)
To cope with complex optimization problems, there exist metaheuristic
techniques, like the evolutionary algorithms.
Genetic Algorithms (GAs) [5] are:
inspired by evolutionary biology;
general purpose randomized optimization techniques;
based on simple rules.
Their basic idea is to evolve iteratively a population of chromosomes, each
representing a candidate solution to the considered problem.
In complex problems requiring the simultaneous optimization of multiple
objectives, GAs are particularly indicated for they deal simultaneously with
a population of solutions.
Advantages:
little information about the problem required;
robust to local optima;
optimization of real, integer and binary problems.
15
[5] L. Chambers, The Practical Handbook of Genetic Algorithms. New York: Champan & Hall, 2001.
15. GA – Principal Steps
1. Initial population of M chromosomes is
generated;
2. The goodness of each chromosome is
evaluated according to predefined
fitness functions;
3. Successively, the GA favors the selection
of the best chromosomes and removes
the others;
4. In the next step a new population is
represented by adopting genetic
operators, e.g., crossover and mutation;
5. All these steps are iterated until a
predefined condition is satisfied.
16
Condition
satisfied?
Nextiteration
Y
N
16. GA – Setup
Idea: exploit the GA capabilities for solving the L0 norm problem.
The NP-hard problem is:
Chromosome structure:
The number of chromosomes M must be fixed;
αi is a chromosome which contains w genes with real values;
The length w of the chromosome is thus equal to the one of the dictionary D.
17
Gene 1 Gene 2 Gene w
…i1
…i2 iw
Chromosome i
Fitness functions:
Multiple objective optimization [6] →
01 min αf
2
22 min xDf α
NSGA-2
αα Dxsubject tomin 0
(1)
[6] K. Deb, Multi-Objective Optimization Using Evolutionary Algorithms. Chichester, U.K.: Wiley, 2001.
17. Experimental Dataset
Aim: to compare the results obtained by the CS reconstructions
with:
Multiresolution Inpainting (MRI) [7]
Contextual Multiple Linear Prediction (CMLP) [8]
In order to quantify the reconstruction accuracy:
1. consider I(1) cloud-free
2. simulate a presence of clouds by obscuring partly I(2)
3. compare the reconstructed I(2) with its original one
Evaluate the sensitivity to two aspects:
Test 1: kind of ground cover obscured
Test 2: size of the contaminated area
20
[7] L. Lorenzi, F. Melgani and G. Mercier, “Inpainting Strategies for Reconstruction of Missing Data in VHR Images”, IEEE Geosci. Remote Sens.
Letters, vol. 8, no. 5, pp. 914-918, Sep. 2011.
[8] F. Melgani, “Contextual Reconstruction of Cloud-Contaminated Multitemporal Multispectral Images”, IEEE Trans. Geosci. Remote Sens., vol. 44,
no. 2, pp. 442–455, Feb. 2006.
18. Multiresolution Inpainting (MRI)
Based on the Region-based inpainting (RBI) from Criminisi et al [9]
21
[7] L. Lorenzi, F. Melgani and G. Mercier, “Inpainting Strategies for Reconstruction of Missing Data in VHR Images”, IEEE Geosci. Remote Sens.
Letters, vol. 8, no. 5, pp. 914-918, Sep. 2011.
[9] A. Criminisi, P. Perez and K. Toyama, “Region filling and object removal by exemplar-based image inpainting”, IEEE Trans. on Image Process.,
vol. 13, no. 9, pp. 1-14, Sep 2004.
In [7], we proposed a processing scheme which
recursively injects multiresolution information for
a better reconstruction of the missing area.
19. Contextual Multiple Linear Prediction
Contextual Multiple Linear Prediction
(CMLP) [8] :
Training:
Prediction:
22
A simple solution is based on the
minimum square error pseudoinverse
technique.
[8] F. Melgani, “Contextual Reconstruction of Cloud-Contaminated Multitemporal Multispectral Images”, IEEE Trans. Geosci. Remote Sens., vol. 44, no. 2,
pp. 442–455, Feb. 2006.
20. Test 1
Contamination of Different Ground Cover
We suppose multiple land cover contamination, namely
different areas of the image are missing
Each mask is composed by ~2,000 pixels
23
21. Test 1
Contamination of Different Ground Cover
Dictionary D is composed by pixels regularly sub-
sampled from region:
316 for DS1
402 for DS2
24
D Sub-sampling
22. Test 1 – Datasets 1 & 2
26
Mask A Mask B Mask C
PSNR Complexity Time [s] PSNR Complexity Time [s] PSNR Complexity Time [s]
MRI 22.54 - 2856 16.05 - 2517 33.77 - 2898
CMLP 20.99 1 1 20.11 1 1 24.05 1 1
OMP 23.96 3 4 20.60 3 4 31.97 3 4
BP 22.22 294 66 24.74 168 59 30.67 301 60
GA 23.78 148 68621 23.15 95 26312 32.01 138 43193
Mask A Mask B
PSNR Complexity Time [s] PSNR Complexity Time [s]
MRI 24.27 - 2995 29.54 - 3614
CMLP 24.61 1 1 27.69 1 1
OMP 26.36 3 5 30.43 3 5
BP 26.45 338 61 31.63 365 91
GA 26.72 173 69231 31.28 201 38475
Dataset 1
Dataset 2
23. Test 1 – Dataset 1, Mask B
28
Reconstructed by OMP Reconstructed by BP Reconstructed by GA
Original crop of I(2) Reconstructed by MRI Reconstructed by CMLP
24. Test 2
Contamination with Different Sizes
We increase the amount of missing data:
mask 1: ~2,000 pixels
mask 2: ~6,000 pixels
mask 3: ~12,000 pixels
Dictionary D is composed as in Test 1
29
26. Test 2 – Dataset 2, Mask 3
31
Reconstructed by OMP Reconstructed by BP Reconstructed by GA
Original crop of I(2) Reconstructed by MRI Reconstructed by CMLP
27. A COMPLETE PROCESSING CHAIN FOR
SHADOW DETECTION AND RECONSTRUCTION
IN VHR IMAGES
PUBLISHED IN THE IEEE TRANSACTION ON GEOSCIENCE AND REMOTE SENSING, VOL.
50, NO. 9, pp. 3440–3452, SEPTEMBER 2012
L. Lorenzi, F. Melgani, and G. Mercier
28. Introduction
In VHR optical images the presence of shadows partially
obscures the scenario.
Missing information directly influences common processing and
analysis operations (e.g. classification).
Shadows can be divided in two classes:
self shadow;
cast shadow.
Hypothesis: shadow and non-shadow classes follow a Gaussian
distribution → more simple and more fast.
Denoting the shadow class as and the non-shadow
class as , the reconstruction of the shadow class will
be reduced to a simple random variable transformation:
33
2
,~ SS
NY
22
,~ˆ,~ SSSS NYNX
,~ 2
SSNX
29. General Block Diagram
34
Supervised
classification
of shadow areas
Original
Image
Restored
Image
Morphological
filtering
Shadow
reconstruction
Border
interpolation
Border
creation
Supervised
classification
of non-shadow
areas
Supervised
shadow vs
non-shadow
classification
I M1 M2 MF
C
Post-
classification &
quality control
30. Step1 - Binary Classification
Separate between shadow and non-shadow areas in
the given image:
Supervised classifier: Support Vector Machine (SVM).
Human help: ROIs collection for Training set.
Feature space defined by:
original image spectral bands;
one-level stationary wavelet transform applied on each spectral
band.
We adopt a Gaussian kernel:
35
Mask M1Original image
2
2121 exp, xxxxKRBF
31. Step2 - Morphological filtering
It is usually adopted to eliminate “salt and pepper”
noise in the images.
We use an opening by reconstruction (erosion (ε)
followed by dilation (δ) ) followed by a closing by
reconstruction (dilation followed by erosion) to
obtain M2.
The size of the structuring element (SE) is 3x3
36
Mask M1 Mask M2
32. Step3 - Border Creation
The transition in-between shadow and non-shadow areas can raise
problems:
e.g., boundary ambiguity, color inconstancy, and illumination variation.
Indeed, the presence of the penumbra induces mixed pixels which
are difficult to classify.
We create a border also adopting morphological operators:
Note that the border is not needed in all directions, but only where
we have the penumbra.
38
22 MMB
Mask M2 Mask MF
33. Step4 - Multiclass Classification
At this step, shadow and non-shadow areas are classified:
the previously mask MF is here exploited to guide the
classification;
note that the same ROIs are adopted here;
to improve map C, a 3x3 majority filter is adopted.
39
Mask MF
Original image Non-shadow classes
Shadow classes
Multiclass
classification
Map C
Post-classification
map
34. Step5 - Quality Control
After the classification, both user (commission errors or
inclusions) and producer (omission errors or exclusions)
accuracies are derived (UA and PA).
UA: an error of commission results when a pixel is committed to an
incorrect class;
PA: an error of omission results when a pixel is incorrectly
classified into another category.
If one of these is lower than a predefined value (80%), it
means that the classification of the corresponding thematic
class is considered of low quality.
In such case, the related shadow compensation is not performed.
40
35. Step6 - Border Reconstruction
In order to smooth the contrast between non-shadow and reconstructed shadow
areas, we implement a contextual linear interpolation.
Along a given direction, a simple affine transformation is applied:
To estimate m and q, we will make use of the least square estimator (pseudo-inverse
method):
This has be done for each available direction.
41
qimz
ZZ
q
m
i
i
z
z
T
NN
1
11
ˆ
1
1..
1
...
36. Dataset Description
Boumerdès (Algeria) Atlanta (USA) Jeddah (Saudi Arabia)
QuickBird
4 spectral bands
0.6 m of resolution
Acquired 28th Feb 2008
16% of shadow cover
IKONOS-2
3 spectral bands
1 m of resolution
Acquired 1998
42% of shadow cover
IKONOS-2
3 spectral bands
1 m of resolution
Acquired 11th Apr 2004
15% of shadow cover
42
44. Experimental Results – Final Results
Small areas still remain dark
(shaded) since they have not
been reconstructed (i.e., UA and
PA of sidewalks class <80%).
The remaining reconstructed
shadow areas look very realistic.
The reconstruction process was
capable to reproduce:
spectral properties;
textural properties.
50
45. Experimental Results – Final Results
Here the results are more mitigated, mainly because of
the darkness and the heterogeneity of the shadow
regions.
Misclassification errors or false alarms, e.g.:
1. a dark roof of a building in the top of the image;
2. the self shadows of some buildings on the left
part of the image;
3. the shadow on the roof in the center of the
image.
Intrinsic complexity of some classes, e.g., the
asphalt class:
Both accuracies (UA and PA) are high, but the
reconstructed shadows appear noisy; the cause
seems to be the multimodal nature of this class.
However, some shadow regions are well reconstructed,
like the bright roofs in the bottom part of the image.
51
46. Experimental Results – Final Results
The Jeddah image is instead
marked by a lot of thin shadows
mostly located in vegetation
areas.
The shadow thinness is explained
by the steepness of the sun
angle at the local image
acquisition hour (11:17 a.m.).
Here the reconstruction, which
was limited to the shaded
vegetation, was globally
satisfactory.
52
47. Impact on Classification Accuracy
54
76 95 74 0(*) 81 78 15(*) 58 0(*) 55(*)
Symbol (*) means that the shadow class was not reconstructed.
49. Conclusion
In this presentation, missing data problems on very high spatial resolution (VHR)
optical satellite have been investigated, in order to detect and/or reconstruct
obscured areas. In particular, we face the problem of:
a complete obscuration due to the presence of cloudy areas;
partial contamination associated with shadow regions.
Inpainting strategies (contribution 1):
independent from the sensor type and from its spatial, temporal and
spectral properties;
completely unsupervised;
sensitivity to the size of the missing area.
Regression with kernel combination (contribution 2):
the fusion of different types of information performed by means of a kernel
combination have made the process particularly promising;
higher (but still contained) computation time.
59
50. Conclusion
Compressive sensing strategies (contribution 3):
good results in the reconstruction of missing areas;
they do not depend on the size of missing area;
completely unsupervised;
sensitivity to the kind of the missing area.
Shadow detection and compensation (contribution 4):
realistic shadow-free images with a promising preservation of spectral and
textural properties;
in case of multimodal non-shadow classes, the implemented linear
compensation method can be found inappropriate.
Shadow reconstructability (contribution 5):
angular second-moment difference, homogeneity difference and variance
ratio behave better than the others;
fusing from 8 to 2 criteria, proved to be a useful way to capture the
synergies between the different criteria;
subjective assessment of the reconstructability of shadow areas.
60
51. Future works
Since three years of research are never enough, future works about
the problem of missing data can understandably be envisioned. For
example:
regarding the inpainting, it could be interesting to integrate the
temporal dimension in the process of reconstruction;
about the compressive sensing approach:
we think that it could be interesting to opt for smart techniques to design the
dictionary, focusing on the context of the missing area;
another possible development is the addition of features in the training step
of the CS (e.g., Haralick textures, Hu invariant moments, …);
regarding the shadow area reconstruction, we may envision to resort to
nonlinear method in case linear regression does not work in a proper
way.
In general, for each of the works presented here, it would be
particularly interesting to reformulate or adapt it in view of an
application for large-scale images knowing that such images raise
the problem of spatial variance.
61
53. List of Related Publications
Journal papers
[J.1] L. Lorenzi, F. Melgani and G. Mercier, “Inpainting strategies for reconstruction of missing
data in VHR images,” IEEE Geosci. Remote Sens. Lett., vol. 8, no. 6, pp. 914–918, Sep.
2011.
[J.2] L. Lorenzi, F. Melgani and G. Mercier, “A complete chain for shadow detection and
reconstruction in VHR images,” IEEE Trans. Geosci. Remote Sens., vol. 50, no. 9, pp.
3440–3452, Sep. 2012.
[J.3] L. Lorenzi, F. Melgani and G. Mercier, “A support vector regression with kernel
combination for missing data reconstruction,” IEEE Geosci. Remote Sens. Lett., in press.
[J.4] L. Lorenzi, F. Melgani, G. Mercier and Y. Bazi, “Assessing the reconstructability of shadow
areas in VHR images,” IEEE Trans. Geosci. Remote Sens., in press.
[J.5] L. Lorenzi, F. Melgani and G. Mercier, “Missing area reconstruction in multispectral images
under a compressive sensing perspective,” IEEE Trans. Geosci. Remote Sens., vol. 51, in
press, 2013.
Conferences
[C.1] L. Lorenzi, F. Melgani and G. Mercier, “Multiresolution inpainting for reconstruction of
missing data in VHR images,” in Proc. IGARSS, Vancouver, Canada, Jul. 2011.
[C.2] L. Lorenzi, F. Melgani and G. Mercier, “Orthogonal matching pursuit for VHR image
reconstruction,” in Proc. IGARSS, Munich, Germany, Jul. 2012.
[C.3] L. Lorenzi, F. Melgani and G. Mercier, “Some criteria to assess the reconstructability of
shadow areas,” in Proc. IGARSS, Munich, Germany, Jul. 2012.
63
