This document describes a method for pixel-level image fusion using principal component analysis (PCA). PCA is used to transform correlated image pixels into a set of uncorrelated principal components. The first principal component accounts for the most variance in the pixel values. To fuse images, the pixels of the input images are arranged into vectors and subtracted from their mean. PCA is applied to get the eigenvectors corresponding to the largest eigenvalues. The normalized eigenvectors are used to compute a fused image as a weighted sum of the input images. Performance is evaluated using metrics like standard deviation, entropy, cross-entropy, and fusion mutual information, with higher values of these metrics indicating better quality of the fused image.

1. Image Fusion Using Principal Component Analysis
Method
Vinit Anil Patil Vijaykumar Vilas Kulkarni
13MCE0088, Communication Engineering, 13MCE0062, Communication Engineering
SENSE, VIT University, Vellore, India SENSE, VIT University, Vellore, India
vinit.anilmalini2013@vit.ac.in kulkarni.vijaykumar2013@vit.ac.in
Abstract— Information fusion can be achieved at any level of
the image information representation- Signal (pixel), feature
(object) and decision (symbol). In this project, we are
concentrating on the Pixel-level image fusion which defines the
process of fusing visual information from a number of
registered images into a single fused image. There are many
image fusion methods such as intensity-hue-saturation, Brovey
transform, principal component analysis (PCA), high-pass
filtering, high-pass modulation, wavelet transform etc. In this
paper, Principal Component Analysis method is used for pixel-
level Image Fusion which basically transforms a number of
Correlated variables into a number of uncorrelated variables.
Performance metrics are used to evaluate the performance of
PCA method with simple average based and wavelet transform
based image fusion algorithm.
Index Terms—Multi-sensor Image Fusion (MIF),Principal
Component Analysis (PCA)
I. INTRODUCTION
Multi-sensor image fusion (MIF) is a technique to
combine the registered images to increase the spatial resolution
of acquired low detail multi-sensor images and preserving their
spectral information. The benefiting fields from MIF are:
Military, remote sensing, machine vision, robotic and medical
imaging, etc. Main objectives of Image Fusion using Principal
Component Analysis are:
(a) The fusion process should preserve all relevant information
contained in the source images,
(b) The fusion process should not introduce any inconsistencies,
and
(c) Irrelevant features and noise should be suppressed to a
maximum extent.
The problem that MIF tries to solve is to merge the
information content from several images (or acquired from
different imaging sensors) taken from the same scene in order to
accomplish a fused image that contains the finest information
coming from the original images. Hence, the fused image would
provide enhanced superiority image than any of the original
source images.
II. LITERATURE REVIEW
A. Image Fusion
Pixel-level image fusion defines the process of fusing
visual information from a number of registered images into a
single fused image. It is part of the much broader subject of
multi sensor information fusion which has attracted a
considerable amount of research attention in the last two
decades.
Multi-sensor information fusion utilizes information
obtained from a number of different sensors surveying an
environment. The aim is to achieve better situation assessment
and/or more rapid and accurate completion of a pre-defined task
than would be possible using any of the sensors individually. In
the literature, it has been defined as the synergistic combination
of different sources of sensory information into a single
representational format The only formal definition of
information fusion (data fusion) to date, is that given by the
U.S. Department of Defence, Joint Directors of Laboratories
Data Fusion Subpanel which represents the first formal body
explicitly dealing with the process of data fusion. Their
definition can be found in as: a multilevel, multifaceted process
dealing with the automatic detection, association, correlation,
estimation and combination of data and information from
multiple sources.
Image fusion represents a specific case of multi sensor
information fusion in which all the information sources used
represent imaging sensors. Information fusion can be achieved
at any level of the image information representation. Analogous
to other forms of information fusion, image fusion is usually
performed at one of the three different processing levels: signal,
feature and decision. Signal level image fusion, also known as
pixel-level image fusion, represents fusion at the lowest level,
where a number of raw input image signals are combined to
produce a single fused image signal. Object level image fusion,
also called feature level image fusion, fuses feature and object
labels and property descriptor information that have already
been extracted from individual input images. Finally, the
highest level, decision or symbol level image fusion represents
fusion of probabilistic decision information obtained by local
decision makers operating on the results of feature level
processing on image data produced from individual sensors.

2. B.Principal Component Analysis
B.1.Theory:
The PCA involves a mathematical procedure that
transforms a number of correlated variables into a number of
uncorrelated variables called principal components. It computes
a compact and optimal description of the data set. The first
principal component accounts for as much of the variance in the
data as possible and each succeeding component accounts for as
much of the remaining variance as possible. First principal
component is taken to be along the direction with the maximum
variance. The second principal component is constrained to lie
in the subspace perpendicular of the first. Within this subspace,
this component points the direction of maximum variance. The
third principal component is taken in the maximum variance
direction in the subspace perpendicular to the first two and so
on. The PCA is also called as Karhunen-Loeve transform or the
Hotelling transform .The PCA does not have a fixed set of basis
vectors like FFT, DCT and wavelet etc. and its basis vectors
depend on the data set.
Let X be a d-dimensional random vector and
assume it to have zero empirical mean. Orthonormal projection
matrix V would be such that Y =VT
X with the following
constraints. The covariance of Y, i.e., cov(Y) is a diagonal and
inverse of V is equivalent to its transpose ( V-1
=VT
). Using
matrix algebra:
Cov (Y) = E {YYT
}
= E {(VT
X) (VT
X)T
}
= E {(VT
X) (XT
V)}……………….(1)
= VT
E {XXT
} V
= VT
cov(X) V
Multiplying both sides of Equation (1) by V, one gets,
V cov(Y) =VVT
cov(X ) V = cov(X )V…………………. (2)
One could write V as V= [V1,V2,………..Vd] and
cov(y) 
1
1
0 0
0 0
0 0
0 0
d
d




 
 
 
 
 
  

 


Substituting Equation (1) into the Equation (2) gives
[λ1 V1 , λ2V2 ……. λd Vd ] =
[cov(X)V1 , cov(X)V2, ……. cov(X)Vd ]
This could be rewritten as
λVi =cov(X) Vi
Where i =1 ,2,...,d and Vi is an eigenvector of cov(X ).
B.2.PCA Algorithm:
Let the source images (images to be fused) be arranged in two-
column vectors.
The steps followed to project this data into a 2-D subspaces are:
1. Organize the data into column vectors. The resulting matrix Z
is of dimension 2 x n.
2. Compute the empirical mean along each column .The
empirical mean vector Me has a dimension of 1 x 2.
3. Subtract the empirical mean vector Me from each column of
the data matrix Z. The resulting matrix X is of dimension 2 x n.
4. Find the covariance matrix C of X i.e. C=XX T mean of
expectation = cov(X)
5. Compute the eigenvectors V and eigenvalue D of C and sort
them by decreasing eigenvalue. Both V and D are of dimension
2 x 2.
6. Consider the first column of V which corresponds to larger
eigenvalue to compute P1 and P2 as:
P1 = V(1) and P2 = V(2)
∑V ∑V
B.3.Image Fusion Using PCA
The information flow diagram of PCA-based image
fusion algorithm is shown in Fig. 1. The input images (images
to be fused) I1 (x, y) and I2 (x, y) are arranged in two column
vectors and their empirical means are subtracted. The resulting
vector has a dimension of n x 2, where n is length of the each
image vector.
Compute the eigenvalues and eigenvectors for this
resulting vector are computed and the eigenvectors
corresponding to the larger eigenvalue are obtained. The
normalized components P1 and P2 (i.e., P1 + P2 = 1) are
computed from the obtained eigenvector. The fused image is:
If(x,y) = P1I1(x,y)+ P2I2(x,y).
Fig.1 Block Diagram of Image Fusion using PCA
III.PERFORMANCE ANALYSIS
Metrices for performance evaluation:
1.Standard deviation-
a) Features-It is known that standard deviation is composed of
the signal and noise parts. This metric would be more efficient
in the absence of noise. It measures the contrast in the fused
image. An image with high contrast would have a high standard
deviation.

3. b) Formula-
 
2
0
(i)
f
L
I
i
i i h

  ,
0
f
L
I
i
i ih

 
where (i)
fIh is the normalized histogram of the Fused image.
(x,y)fI and L number of frequency bins in the histogram.
2.Entropy-
a) Features- Entropy is used to measure the information content
of an image. Entropy is sensitive to noise and other unwanted
rapid fluctuations. An image with high information content
would have high entropy.
b) Formula- Using the entropy, the information content of a
fused image is:
2
0
(i)log (i)
f f
L
I I
i
He h h

 
3. Cross entropy-
a)Features-Cross-entropy evaluate the similarity in information
content between input images and fused image. Fused and
reference images containing the same information would have a
low cross entropy.
b)Formula is given by-
1
1
2
2
1 2
1 2
1
0
2
0
(I ;I ) CE(I ;I )
(I ,I ;I )
2
(i)
(I ;I ) (i)log
(i)
(i)
(I ;I ) (i)log
(i)
f
f
f f
f
L
I
If
Ii
L
I
If
Ii
CE
CE
h
CE h
h
h
CE h
h


 
 
 
 
 
 
 
 
 
 






4. Spatial frequency-
a) Features- This frequency in spatial domain indicates the
overall activity level in the fused image.
b) Formula-
Spatial Frequency criterion SF is: 2 2SF RF CF 
Where Row Frequency of the image:
2
1 1
1
(x, y) I (x, y 1)
M N
f f
x y
RF I
MN  
 
  
   
Column Frequency of the image:
2
1 1
1
(x, y) I (x 1, y)
N M
f f
y x
CF I
MN  
 
  
   
5. Fusion mutual information-
a) Features-It measures the degree of dependence of the two
images. A larger measure implies better quality.
b) Formula-
If the joint histogram between 1(x,y)I and (x,y)fI is defined
as
1
(i, j)
fI Ih and 2 (x, y)I and (x,y)fI as
2
(i, j)
fI Ih . Then
the mutual information between source and fused images are:
1 1f fI I I IFMI MI MI  where
1
1 1
1
2
2 2
2
2
1 1
2
1 1
(i, j)
(i, j) log
(i, j) (i, j)
(i, j)
(i, j) log
(i, j) (i, j)
f
f f
f
f
f f
f
M N I I
I I I I
I Ii j
M N I I
I I I I
I Ii j
h
M I h
h h
h
M I h
h h
 
 
 
 
 
 
 
 
 
 
 
 


 
 
6. Fusion quality index-
a) Features-The range of this metric is 0 to 1. One indicates the
fused image contains all the information from the source
images.
) Formula-
  1 2(w) (w)QI(I ,I | w) 1 (w) (I ,I | w)f f
w W
FQI c QI 

  
where 1
1 2
2
2 2
(w)
I
I I


 


computed over a window;
1 2
2 2(w) MAX( , )I IC   over a window (w)c is a
normalized version of (w)C and 1QI(I ,I | w)f is the quality
index over a window for a given source image and fused image.
7. Fusion similarity metric-
a) Feature-It takes into account the similarity between the
source and fused image block within the same spatial position.
The range of this metric is zero to one. The value one indicates
that the fused image contains all the information from the
source images.
b)Formula-
1
1 2 2
2
QI(I ,I | w)
(I , I ,I | w) (I , I | w)
(I ,I | w)
f
f f
fw W
FSM sim QI
QI
 
 
 
 
 

where
1 2(I ,I ,I | w) 0fsim  if
1
1 2
0f
f f
I I
I I I I

 


=
1
1 2
f
f f
I I
I I I I

 
if
1
1 2
0 1f
f f
I I
I I I I

 
 

= 1 if
1
1 2
1f
f f
I I
I I I I

 



4. IV.RESULT
The aircraft, shown in Fig.2 is considered as a
reference image to evaluate the performance of the fusion
algorithm. The complementary pair input images and are taken
to evaluate the fusion algorithm and these images are shown in
Fig. 3 and 4. The images are complementary in the sense that
the blurring occurs at the left-half and the right half
respectively. The error (difference) image (Fig.6) is computed
by taking the corresponding pixel difference
of reference mage and fused image ,i.e.,
IE(x, y) = IR (x, y) – IF (x, y)
The fused and error images by PCA algorithm are
shown in Fig.5 & 6.
Figure 2. Reference image (x, y)rI
Figure 3. Source image1 1(x, y)I
Figure 4. Source image2 2 (x, y)I
Figure 5. Fused image IF (x, y)
Figure 6. Error image IE(x, y)

5. V.OBSERVATION & CONCLUSION
Pixel-level image fusion using principal component
analysis is implemented in PC MATLAB. Different image
fusion performance metrics have been evaluated. The simple
averaging fusion algorithm shows degraded performance.
Image fusion using the PCA shows better performance.
Algorithm used He SD SF
PCA 6.2897 45.4530 7.6082
DWT 6.2925 45.6415 11.0283
VI. FUTURE SCOPE
Future of image fusion will be to develop such
techniques that will automatically combine images of a scene
captured under different illumination. Beyond providing digital
tools for artists for creating surrealist images and videos, the
methods can also be used for practical applications. For
example, the non-realistic appearance can be used to enhance
the context of night time traffic videos so that they are easier to
understand. The context is automatically captured from a fixed
camera and inserted from a day-time image (of the same scene).
Our approach is based on a gradient domain technique that
preserves important local perceptual cues while avoiding
traditional problems such as aliasing, ghosting and haloing.
VII. REFERENCES
[1] Zhijun Wang, Djemel Ziou, Costas Armenakis, Deren Li,
and Qingquan Li,”A Comparative Analysis of Image Fusion
Methods” IEEE TRANSACTIONS ON GEOSCIENCE AND
REMOTE SENSING, VOL. 43, NO. 6, JUNE 2005.
[2] Digital Image Processing by Gonzalez & Woods.
[3] Digital Image Processing using MATLAB by Gonzalez &
Woods.