A short brief of distance and statistical distance which is core of multivariate analysis.................you will get here some more simple conception about distances and statistical distance.
2. Md. Menhazul Abedin
M.Sc. Student
Dept. of Statistics
Rajshahi University
Mob: 01751385142
Email: menhaz70@gmail.com
3. Objectives
β’ To know about the meaning of statistical
distance and itβs relation and difference with
general or Euclidean distance
4. Content
οΆDefinition of Euclidean distance
οΆConcept & intuition of statistical distance
οΆDefinition of Statistical distance
οΆNecessity of statistical distance
οΆConcept of Mahalanobis distance (population
&sample)
οΆDistribution of Mahalanobis distance
οΆMahalanobis distance in R
οΆAcknowledgement
9. we see that two specific points in each picture
Our problem is to determine the length between
two points .
But how ??????????
Assume that these pictures are placed in two
dimensional spaces and points are joined by a
straight line
10. Let 1st point is (π₯1,π¦1) and 2nd point is (π₯2, π¦2)
then distance is
D= β ( (π₯1βπ₯2)2
+ (π¦1 β π¦2)2
)
What will be happen when dimension is three
12. Distance is given by
β’ Points are (x1,x2,x3) and (y1,y2,y3)
(π₯1 β π¦1)2+(π₯2 β π¦2)2+(π₯3 β π¦3)2
13. For n dimension it can be written
as the following expression and
named as Euclidian distance
22
22
2
11
2121
)()()(),(
),,,(),,,,(
pp
pp
yxyxyxQPd
yyyQxxxP
οο«ο«οο«οο½ ο
οο
14. 12/12/2016 14
Properties of Euclidean Distance and
Mathematical Distance
β’ Usual human concept of distance is Eucl. Dist.
β’ Each coordinate contributes equally to the distance
22
22
2
11
2121
)()()(),(
),,,(),,,,(
pp
pp
yxyxyxQPd
yyyQxxxP
οο«ο«οο«οο½ ο
οο
14
Mathematicians, generalizing its three properties ,
1) d(P,Q)=d(Q,P).
2) d(P,Q)=0 if and only if P=Q and
3) d(P,Q)=<d(P,R)+d(R,Q) for all R, define distance
on any set.
17. β’ The Manhattan distance is the simple sum of
the horizontal and vertical components,
whereas
the diagonal distance might be computed by
applying the Pythagorean Theorem .
19. β’ Manhattan distance 12 unit
β’ Diagonal or straight-line distance or Euclidean
distance is 62 + 62 =6β2
We observe that Euclidean distance is less than
Manhattan distance
23. Relationship between Manhattan &
Euclidean distance.
β’ It now seems that the distance from A to C is 7 blocks,
while the distance from A to B is 6 blocks.
β’ Unless we choose to go off-road, B is now closer to A
than C.
β’ Taxicab distance is sometimes equal to Euclidean
distance, but otherwise it is greater than Euclidean
distance.
Euclidean distance <Taxicab distance
Is it true always ???
Or for n dimension ???
27. For high dimension
β’ It holds for high dimensional case
β’ Ξ£ βπ₯π β π¦πβ2
β€ Ξ£ βπ₯π β π¦πβ2
+ 2Ξ£βπ₯π β π₯πββπ₯π β π₯πβ
Which implies
Ξ£ (π₯π β π¦π)2 β€ Ξ£βπ₯π β π₯πβ
π πΈ β€ π π
28. 12/12/2016
Statistical Distance
β’ Weight coordinates subject to a great deal of
variability less heavily than those that are not
highly variable
Whoisnearerto
datasetifitwere
point?
Same
distance from
origin
29. β’ Here
variability in x1 axis > variability in x2 axis
ο± Is the same distance meaningful from
origin ???
Ans: no
But, how we take into account the different
variability ????
Ans : Give different weights on axes.
31. all point that have coordinates (x1,x2) and
are a constant squared distance , c2
from the
origin must satisfy
π₯12
π 11
+
π₯22
π 22
=π2
But β¦ how to choose c ?????
Itβs a problem
Choose c as 95% observation fall in this area β¦.
π 11 > π 22
= >
1
π 11
<
1
π 22
33. β’ This expression can be generalized as β¦β¦β¦
statistical distance from an arbitrary point
P=(x1,x2) to any fixed point Q=(y1,y2)
;lk;lk;
For P dimensionβ¦β¦β¦β¦β¦..
34. Remark :
1) The distance of P to the origin O is
obtain by setting all π¦π = 0
2) If all π ππ are equal Euclidean
distance formula is appropriate
36. β’ How do you measure the statistical distance of
the above data set ??????
β’ Ans : Firstly make it uncorrelated .
β’ But why and howβ¦β¦β¦???????
β’ Ans: Rotate the axis keeping origin fixed.
41. Choice of π
οΆWhat π will you choice ?
οΆHow will you do it ?
οΆ Data matrix β Centeralized data matrix β Covariance of
data matrix β Eigen vector
οΆTheta = angle between 1st eigen vector and [1,0]
or
angle between 2nd eigen vector and [0,1]
42. Why is that angle between 1st eigen vector and
[0,1] or angle between 2nd eigen vector and [1,0]
??
Ans: Let B be a (p by p) positive definite matrix
with eigenvalues Ξ»1β₯Ξ»2β₯Ξ»3β₯ β¦ β¦ . . β₯ Ξ»p>0
and associated normalized eigenvectors
π1, π2, β¦ β¦ β¦ , π π.Then
πππ₯ π₯β 0
π₯β² π΅π₯
π₯β² π₯
= Ξ»1 attained when x= π1
πππ π₯β 0
π₯β² π΅π₯
π₯β² π₯
= Ξ» π attained when x= π π
44. Choice of π
#### Excercise 16.page(309).Heights in inches (x) &
Weights in pounds(y). An Introduction to Statistics
and Probability M.Nurul Islam #######
x=c(60,60,60,60,62,62,62,64,64,64,66,66,66,66,68,
68,68,70,70,70);x
y=c(115,120,130,125,130,140,120,135,130,145,135
,170,140,155,150,160,175,180,160,175);y
############
V=eigen(cov(cdata))$vectors;V
as.matrix(cdata)%*%V
plot(x,y)
49. β’ ################ comparison of both
method ############
comparison=tdata -
as.matrix(cbind(xx,yy));comparison
round(comparison,4)
50. ########### using package. md from original data #####
md=mahalanobis(data,colMeans(data),cov(data),inverted =F);md
## md =mahalanobis distance
######## mahalanobis distance from transformed data ########
tmd=mahalanobis(tdata,colMeans(tdata),cov(tdata),inverted =F);tmd
###### comparison ############
md-tmd
51. Mahalanobis distance : Manually
mu=colMeans(tdata);mu
incov=solve(cov(tdata));incov
md1=t(tdata[1,]-mu)%*%incov%*%(tdata[1,]-
mu);md1
md2=t(tdata[2,]-mu)%*%incov%*%(tdata[2,]-
mu);md2
md3=t(tdata[3,]-mu)%*%incov%*%(tdata[3,]-
mu);md3
............. β¦β¦β¦β¦β¦. β¦β¦β¦β¦..
md20=t(tdata[20,]-mu)%*%incov%*%(tdata[20,]-
mu);md20
md for package and manully are equal
59. β’ The above distances are completely
determined by the coefficients(weights)
πππ ; i, k = 1,2,3, β¦ β¦ β¦ p. These are can be
arranged in rectangular array as
this array (matrix) must be symmetric positive
definite.
60. Why Positive definite ????
Let A be a positive definite matrix .
A=CβC
XβAX= XβCβCX = (CX)β(CX) = YβY It obeys
all the distance property.
XβAX is distance ,
For different A it gives different distance .
61. β’ Why positive definite matrix ????????
β’ Ans: Spectral decomposition : the spectral
decomposition of a kΓk symmetric matrix
A is given by
β’ Where (Ξ»π, ππ); π = 1,2, β¦ β¦ β¦ , π are pair of
eigenvalues and eigenvectors.
And Ξ»1 β₯ Ξ»2 β₯ Ξ»3 β₯ β¦ β¦ . . And if pd Ξ»π > 0
& invertible .
66. β’ Consider the Euclidean distances from the
point Q to the points P and the origin O.
β’ Obviously d(PQ) > d (QO )
ο± But, P appears to be more like the points in
the cluster than does the origin .
ο± If we take into account the variability of the
points in cluster and measure distance by
statistical distance , then Q will be closer to P
than O .
67. Mahalanobis distance
β’ The Mahalanobis distance is a descriptive
statistic that provides a relative measure of a
data point's distance from a common point. It
is a unitless measure introduced by P. C.
Mahalanobis in 1936
68. Intuition of Mahalanobis Distance
β’ Recall the eqution
d(O,P)= π₯β² π΄π₯
=> π2
(π, π) =π₯β²
π΄π₯
Where x=
π₯1
π₯2
, A=
π11 π12
π21 π22
71. Mahalanobis Distance
β’ Mahalanobis used ,inverse of covariance
matrix Ξ£ instead of A
β’ Thus π2
π, π = π₯β²
Ξ£β1
π₯ β¦β¦β¦β¦β¦..(1)
β’ And used π (ππππ‘ππ ππ ππππ£ππ‘π¦ ) instead of y
π2
(π, π) = (π₯ β π )β²Ξ£β1
(π₯ β π)β¦β¦β¦..(2)
Mah-
alan-
obis
dist-
ance
72. Mahalanobis Distance
β’ The above equations are nothing but
Mahalanobis Distance β¦β¦
β’ For example, suppose we took a single
observation from a bivariate population with
Variable X and Variable Y, and that our two
variables had the following characteristics
73. β’ single observation, X = 410 and Y = 400
The Mahalanobis distance for that single value
as:
75. β’ Therefore, our single observation would have
a distance of 1.825 standardized units from
the mean (mean is at X = 500, Y = 500).
β’ If we took many such observations, graphed
them and colored them according to their
Mahalanobis values, we can see the elliptical
Mahalanobis regions come out
76. β’ The points are actually distributed along two
primary axes:
77.
78. If we calculate Mahalanobis distances for each
of these points and shade them according to
their distance value, we see clear elliptical
patterns emerge:
79.
80. β’ We can also draw actual ellipses at regions of
constant Mahalanobis values:
68%
obs
95%
obs
99.7%
obs
81. β’ Which ellipse do you choose ??????
ο±Ans : Use the 68-95-99.7 rule .
1) about two-thirds (68%) of the points should
be within 1 unit of the origin (along the axis).
2) about 95% should be within 2 units
3)about 99.7 should be within 3 units
83. Sample Mahalanobis Distancce
β’ The sample Mahalanobis distance is made by
replacing Ξ£ by S and π by π
β’ i.e (X- π)β πβ1
(X- π)
84. For sample
(X- πΏ)β πΊβπ
(X- πΏ)β€ π π
π (β)
Distribution of mahalanobis distance
85. Distribution of mahalanobis distance
Let π1, π2, π3, β¦ β¦ β¦ , π π be in dependent
observation from
any population with
mean π and finite (nonsingular) covariance Ξ£ .
Then
οΆ π ( π β π) is approximately ππ(0, Ξ£)
and
οΆ π π β π β²
πβ1
( π β π) is approximately Ο π
2
for n-p large
This is nothing but central limit theorem
86. Mahalanobis distance in R
β’ ########### Mahalanobis Distance ##########
β’ x=rnorm(100);x
β’ dm=matrix(x,nrow=20,ncol=5,byrow=F);dm ##dm = data matrix
β’ cm=colMeans(dm);cm ## cm= column means
β’ cov=cov(dm);cov ##cov = covariance matrix
β’ incov=solve(cov);incov ##incov= inverse of
covarianc matrix
87. Mahalanobis distance in R
β’ ####### MAHALANOBIS DISTANCE : MANUALY ######
β’ @@@ Mahalanobis distance of first
β’ observation@@@@@@
β’ ob1=dm[1,];ob1 ## first observation
β’ mv1=ob1-cm;mv1 ## deviatiopn of first
observation from
center of gravity
β’ md1=t(mv1)%*%incov%*%mv1;md1 ## mahalanobis
distance of first
observation from center of
gravity
β’
88. Mahalanobis distance in R
β’ @@@@@@ Mahalanobis distance of second
observation@@@@@
β’ ob2=dm[2,];ob2 ## second observation
β’ mv2=ob2-cm;mv2 ## deviatiopn of second
β’ observation from
β’ center of gravity
β’ md2=t(mv2)%*%incov%*%mv2;md2 ##mahalanobis
distance of second
observation from center of
gravity
................ β¦β¦β¦β¦β¦β¦ β¦..β¦β¦β¦β¦β¦
89. Mahalanobis distance in R
β¦β¦β¦....... β¦β¦β¦β¦β¦β¦ β¦β¦β¦β¦β¦
@@@@@ Mahalanobis distance of 20th
observation@@@@@
β’ Ob20=dm[,20];ob20 [## 20th observation
β’ mv20=ob20-cm;mv20 ## deviatiopn of 20th
observation from
center of gravity
β’ md20=t(mv20)%*%incov%*%mv20;md20
## mahalanobis distance of
20thobservation from
center of gravity
90. Mahalanobis distance in R
####### MAHALANOBIS
DISTANCE : PACKAGE ########
β’ md=mahalanobis(dm,cm,cov,inverted =F);md
## md =mahalanobis
distance
β’ md=mahalanobis(dm,cm,cov);md
91. Another example
β’ x <- matrix(rnorm(100*3), ncol = 3)
β’ Sx <- cov(x)
β’ D2 <- mahalanobis(x, colMeans(x), Sx)