This document discusses vector space interpretation of random variables. It begins by introducing vector spaces and their properties such as closure under addition and scalar multiplication. Random variables can be interpreted as elements of a vector space. Inner products, norms, orthogonality and projections are discussed in the context of both vector spaces and random variables. Interpreting expectations as inner products allows treating random variables as vectors in an inner product space.

1. VECTOR SPACE INTERPRETATION OF RANDOM VARIABLES
Interpretation of random variables as elements of a vector space helps in understanding many
operations involving random variables. We start with an introduction to the concepts of the
vector space. We will discuss the principles of the minimum mean-square error estimation and
the linear minimum mean-square error estimation of a signal from noise and the vector-space
interpretation of the later.
Vector space
Consider a set V with elements called vectors and the set of real numbers . The elements of
will be called scalars. A vector will be denoted by bold-face character like v. Suppose two
operations called vector addition and scalar multiplication respectively on V.
V is called a vector space if the following properties are satisfied.
(1) ( V ,+) is a commutative group. Thus (V ,+) satisfies the following properties.
(i) Closure property: For any pair of elements ,v,w V there exists a unique
element .(v + w) V
(ii) Associativity: Vector addition is associative: v +(w +z) = (v +w)+z for
any three vectors v,w,z V
(iii) Existence of the zero vector: There is a vector 0 V such that
v +0 = 0+ v v for any .v V
(iv) Existence of the additive inverse: For any v V there is a vector  v V such
that
.v +(-v) = 0 = (-v)+ v
(v) Commutativity: For any ,v,w V .v + w = w + v
(2) For any element v V and any .r  the scalar multiplication r v V
This the scalar multiplication has the following properties for any , .r s and any
,v,w V

2. (1) Associativity: ( ) ( ) for , andr s rs r s  v v v V
(2) Distributivity with respect to vector addition: r r r(v + w) = v + w
(3) Distributivity with respect to sacalar addition:
r s r s( )v = v + v
(4) Unity scalar: 1v = v
Example 1:
Let S be an arbitrary set and V be the set of all functions from S to . Suppose :f S  and
:g S  denote two functions and s S .and a . Then by definition,
( )( ) ( ) ( )f g s f s g s   and
( )( ) ( )af s af s .
Therefore, addition of functions and scalar multiplication of a function are function in V. It is
easy to verify that addition of functions and the multiplication of a function by scalar satisfies
the properties of vector space. Particularly, the zero function is a function that maps all
elements in S to the real number 0. Thus,
0( ) 0,s s S  
The random variables defined on a probability space ( , , )S P are functions on the sample space.
Therefore, the set of random variables forms a vector space with respect to the addition of
random variables and scalar multiplication of a random variable by a real number.
Subspace
Suppose W is a non-empty subset of V. W is called a subspace of V if W is a vector space
with respect to the vector addition and scalar multiplication defined on V.
For a non-empty subset W is a subspace of , it is sufficient that W is closed under the vector
addition and the scalar multiplication of V. Thus the sufficient conditions are:
(1) , , and
(2) , ,r r
  
   
v w W (v + w) W
v W v W
Linear Independence and Basis
Consider a subset of n vectors 1 2 nB = {b ,b ,...,b }
If 1 2 .... nc c c   1 2 nb b b 0 implies that
1 2 ... 0,nc c c    then n1 2b ,b ,...,b are called linearly independen (LI)t.

3. The subset 1 2 nB = {b ,b ,...,b } of n LI vectors is called a basis if each v V can be expressed
as a linear combination of elements of B. The number of elements in B is called the dimension
of V. Thus B = {i, j,k} is a basis of 3
.
Norm of a vector
Suppose v is a vector in a vector space V defined over . The norm, v is a scalar such that
andV r  v,w
1. 0
2. 0 only when
3.
4. ( Triangle Inequality )
r r

 

  
v
v v 0
v v
v w v w
A vector space V where norm is defined is called an normed vector space. For example,
following are valid norms of 1 2[ ... ] n
nv v v  v :
(i) 2 2 2
1 2 ... nv v v  v
(ii) 1 2max( , ,..., )nv v vv
Inner Product
If vand w are real vectors in a vector space V defined over , the inner product  v,w is a
scalar such that andV r  v,w,z
2
1.
2. 0, where is a induced by the inner product
3.
4.
norm
,
r , r
    
   
        
    
v, w w, v
v, v v v
v w z v, z w, z
v w v, w
A vector space V where an inner product is defined is called an inner product space. Following
are examples of inner product spaces:
(i) n
with the inner product  v, w =
1 1 2 2 1 2 1 2... , [ ... ] , [ ... ]n n
n n n nv w v w v w v v v w w w         v w v w =
(ii) The space 2
( )L of square-integrable real functions with the inner product
1 2 1 2, ( ) ( )f f f x f x dx


  
2
1 2, ( )f f L 

4. Cauchy Schwarz Inequality
For any two vectors vand w belonging to an inner product space V ,
  v, w w v
Let us consider
2
2 22
then , 0
, 2 , , 0
2 , 0
c c c
c c c
c c
       
      
     
z v w z v w v w
v v v w w w
v v w w
The left-hand side of the last inequality is a quadratic expression in variable c For the above
quadratic expression to be non-negative, the discriminant must be non-positive.. So
2 22
2 22
2 22
| 2 , | 4 0
| , | 0
,| |
| , |
   
    
   
   
v w v w
v w v w
wv v w
v w v w
The equality holds when
0 0c
c
   
 
z v w
v w
Hilbert space
Note that the inner product induces a norm that measures the size of a vector. Thus, v w is a
measure of distance between the vectors ,v,w V We can define the convergence of a
sequence of vectors in terms of this norm.
Consider a sequence of vectors  , 1,2,...,n  nv The sequence is said to converge to a limit
vector v if corresponding to every 0 , we can find a positive integer N such that
for .n N   nv v The sequence of vectors  , 1,2,...,n  nv is said to be a
Cauchy sequence if
,
lim 0 .m
n m 
 nv v
In analysis, we may require that the limit of a Cauchy sequence of vectors is also a member of
the inner product space. Such an inner prduct space where every Cauchy sequence of vectors is
convergent is known as a Hilbert space.
Orthogonal vectors
Two vectors vand w belonging to an inner product space V are called orthogonal if
0  v, w
Orthogonal vectors are independent and a set of n orthogonal vectors 1 2 nB = {b ,b ,...,b } forms a
basis of the n-dimensional vector space.

5. Orthogonal projection
It is one of the important concepts in linear algebra widely used in random signal processing.
Suppose W is a subspace of an inner product space V. Then the subset
{ | , 0 }v v
     W V v,w w W
is called the orthogonal complement of W.
Any vector v in a Hilbert space V can be expressed as
1v = w + w
where 1, 
 w W w W . In such a decomposition, w is called the orthogonal projection of v
on W and represents closest approximation of v by a vector in W in the following sense
min

  
u W
v w v u
We omit the proof of this result. The result can be geometrically illustrated as follows:
Gram-Schimdt orthogonalisation
Joint Expectation as an inner product
Interpreting the random variables andX Y as vectors, the joint expectation EXY satisfies the
properties of the inner product of two vectors. and .X Y Thus
< > XYX,Y = E
We can also define the norm of the random variable X by
2 2
< > XX X, X = E
Similarly, two random variables andX Y are orthogonal if < > XYX,Y = E = 0
We can easily verify that EXY satisfies the axioms of inner product.
The norm of a random variable X is given by
2 2
X = EX
u
1w
v
v
w
v
W
v

6. For two n dimensional random vectors
1
2
n
X
X
X
 
 
 
 
 
 
X and
1
2
,
n
Y
Y
Y
 
 
 
 
 
 
Y the inner product is
1
n
i i
i
< >= E EX Y

  X, Y X Y
The norm of an n-dimensional random vector X is given by
2 2
1
n
i
i
< >= E EX

  X X, X X X
Orthogonal Random Variables and Orthogonal Random Vectors
 Two vectors vand w are called orthogonal if 0  v, w
 Two random variables YX and are called orthogonal if 0.EXY 
 Similarly two n-dimension random vectors andX Y are called orthogonal if
1
0
n
i i
i
E EX Y

  X Y
 Just like the independent random variables and the uncorrelated random variables, the
orthogonal random variables form an important class of random variables.
 If YX and are uncorrelated, then
( )( ) 0
( ) is orthogonal to ( )
X Y
X Y
E X Y
X Y
 
 
  
  
 If each of YX and is of zero-mean
( , )Cov X Y EXY
In this case, 0 ( ) 0.EXY Cov XY  