KALMAN FILTER
APPLICATION IN COMPUTER VISION
IMAGE PROCESSING

School of Electrical & Computer Eng.

Hamid Bazargani

&
Outline:
• A brief introduction to Kalman filter
• Classical Kalman Filter
• Kalman & Stereo vision localization
• Differe...
Introduction:
• Kalman is a recurrent algorithm relying on series of uncertain
measurements to optimally estimate an unkno...
Formulation:

Where F is transition matrix, G is a control input matrix . H is output
matrix.
Random variables w and v are...
Kalman stages
Predict or time update

Correct or measurement update
Multivariate Gaussians:
posterior
prior

Measurement
evidence

Markov model:

p( x | z ) ∝ p( z | x) p( x)
Multivariate Gaussians:
posterior
Measurement
evidence

prior

 1

p ( z | x) ∝ exp − ( x − z )T Γ −1 ( x − z )
 2

...
Multivariate Gaussians:

CS373 - Programming a Robotic Car - Unit 2
Kalman Filter Prediciton - CS373 Unit 2 - Udacity
By S...
Kalman & Stereo Vision Localization:

• Stereo localization requires both camera’s information at each time
step.
Kalman & Stereo Vision Localization:

• There is a small time delay between the two cameras’ image
capturing. This time la...
Non-linearity & Kalman Filter:
[Refer to Video1]

• Python and Blender

used for data acquisition which brings Physics eng...
Non-linearity & Kalman Filter:
Extended Kalman Filter

Figures from CS-223 (Stanford)
Non-linearity & Kalman Filter:
Extended Kalman Filter

Figures from CS-223 (Stanford)
Non-linearity & Kalman Filter:
Extended Kalman Filter

Figures from CS-223 (Stanford)
Adaptive Kalman Filter:
• Adaptive Fading Kalman Filter
• Innovation based Adaptive Kalman Filter
Kalman Filter Categories

Kalman

EKF

Non-linearity

AEKF

AKF

Adaption
SV Results

Noise level σ n = 0.05

Noise level σ n = 0.5
Image and Video de-noising:

• noise is assumed to be signal-independent at each pixel
Image and Video de-noising:
• Wiener Filter
• Bilateral Filter
• Median Filter
• Non-Local Means Filtering
Image and Video de-noising:
• Wiener Filter
• Assume Stationary signal
• Signal and noise are independent with constant PS...
Image and Video de-noising:
• Wiener Filter
Image and Video de-noising:
• Wiener Filter

σ n = 15
Image and Video de-noising:
• Wiener Filter

Filtered
Image and Video de-noising:
• Non-Local Means Filter
• Each pixel is replaced by weighted average of its neighborhoods.
• ...
Image and Video de-noising:
• Non-Local Means Filter

σ n = 15
Image and Video de-noising:
• Non-Local Means Filter
Image and Video de-noising:
• Comparison
original

Wiener filter

Noisy

σ n = 15

NLM filter
Image and Video de-noising:
• Comparison

PSNR (db)
14
12
10
8
6
4
2
0

PSNR (db)
Video de-noising:
• Each pixel is replaced by 3D spatio-temporal weighted average of its
neighborhoods.
Motion estimation & Kalman Filter
• Optical Flow Algorithms
• Block Matching Algorithms
• Full Search Algorithm (FSA)
• Th...
Motion estimation & Kalman Filter
• Why Kalman Filter?
• Motion estimation inaccuracy when illumination is changed
• Inacc...
Motion estimation & Kalman Filter
2D Causal AR model:

B(m,n-1)

B(m-1,n)

B(m,n)

vi , x (m, n) = a01vi , x (m, n − 1) + ...
Motion estimation & Kalman Filter
State space representation :
 vi , x (m, n)  a01 a11 0 0  vi , x (m − 1, n) q 
...
Motion estimation & Kalman Filter
Motion estimation & Kalman Filter
Motion estimation & Kalman Filter
Motion estimation & Kalman Filter
Motion estimation & Kalman Filter
Motion estimation & Kalman Filter
• Green arrows show motion vectors
• Red arrows show corrected motion vectors
• Kalam us...
Motion estimation Results
[Refer to Video2]

• Results are generated using the following approaches:
• 2D 3x3 spatial doma...
Motion estimation Results
[Refer to Video2]
Kalman Tracking for Robust pose estimation
• Object Recognition
• Object recognition requires set of 2D correspondences be...
Kalman Tracking for Robust pose estimation
• Tracking
• Feature points tracking
• Tracking the region of search for matche...
Kalman Tracking for Robust pose estimation
• Pose estimation:
Video
Frame points

Target Points
Point
Matching

RANSAC Loo...
Kalman Tracking for Robust pose estimation
• Pose estimation:
[Refer to Video3]
Implementation
• Programs are written with C++ using OpenCV library
• C++ implementation of generic Kalman Filter algorith...
Conclusion:
• We studied Kalman Filter as an adaptive filter which can be applied for
Smoothing, prediction , filtering an...
References:
1.

Dan Simon. Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches. Wiley-Interscience, 1
e...
Thank you for your Attention.

http://www.slideshare.net/hamidbazargani/venndiagrams-29181371
Upcoming SlideShare
Loading in...5
×

Kalman ip cv

631

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
631
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Kalman ip cv

  1. 1. KALMAN FILTER APPLICATION IN COMPUTER VISION IMAGE PROCESSING School of Electrical & Computer Eng. Hamid Bazargani &
  2. 2. Outline: • A brief introduction to Kalman filter • Classical Kalman Filter • Kalman & Stereo vision localization • Different extensions of Kalman Filter • Image and Video De-noising (Kalman approach) • Kalman Tracking for Robust pose estimation
  3. 3. Introduction: • Kalman is a recurrent algorithm relying on series of uncertain measurements to optimally estimate an unknown variable. • The Kalman filter is 50 years old but is still a popular method for sensory information analysis . • The Kalman applications span area of navigation systems, aerospace, robotics, information processing , computer vision, etc • Main properties: • Requires small computational cost • State space representation and ease of implementation • Recursive algorithm which only uses last state and current observation
  4. 4. Formulation: Where F is transition matrix, G is a control input matrix . H is output matrix. Random variables w and v are process and measurement noise respectively.
  5. 5. Kalman stages Predict or time update Correct or measurement update
  6. 6. Multivariate Gaussians: posterior prior Measurement evidence Markov model: p( x | z ) ∝ p( z | x) p( x)
  7. 7. Multivariate Gaussians: posterior Measurement evidence prior  1  p ( z | x) ∝ exp − ( x − z )T Γ −1 ( x − z )  2   1  p ( x) ∝ exp − ( x − µ )T Σ −1 ( x − µ )  2  µ new = (Σ −1 + Γ −1 ) −1 (Σ −1µ + Γ −1 z ) ( Σ new = Σ −1 + Γ −1 ) −1
  8. 8. Multivariate Gaussians: CS373 - Programming a Robotic Car - Unit 2 Kalman Filter Prediciton - CS373 Unit 2 - Udacity By Sebastian Thrun
  9. 9. Kalman & Stereo Vision Localization: • Stereo localization requires both camera’s information at each time step.
  10. 10. Kalman & Stereo Vision Localization: • There is a small time delay between the two cameras’ image capturing. This time lag causes an error in estimation of object position in Cartesian coordinates.
  11. 11. Non-linearity & Kalman Filter: [Refer to Video1] • Python and Blender used for data acquisition which brings Physics engine to generate exact trajectory. • Blender communicate with matlab over UDP protocol.
  12. 12. Non-linearity & Kalman Filter: Extended Kalman Filter Figures from CS-223 (Stanford)
  13. 13. Non-linearity & Kalman Filter: Extended Kalman Filter Figures from CS-223 (Stanford)
  14. 14. Non-linearity & Kalman Filter: Extended Kalman Filter Figures from CS-223 (Stanford)
  15. 15. Adaptive Kalman Filter: • Adaptive Fading Kalman Filter • Innovation based Adaptive Kalman Filter
  16. 16. Kalman Filter Categories Kalman EKF Non-linearity AEKF AKF Adaption
  17. 17. SV Results Noise level σ n = 0.05 Noise level σ n = 0.5
  18. 18. Image and Video de-noising: • noise is assumed to be signal-independent at each pixel
  19. 19. Image and Video de-noising: • Wiener Filter • Bilateral Filter • Median Filter • Non-Local Means Filtering
  20. 20. Image and Video de-noising: • Wiener Filter • Assume Stationary signal • Signal and noise are independent with constant PSD
  21. 21. Image and Video de-noising: • Wiener Filter
  22. 22. Image and Video de-noising: • Wiener Filter σ n = 15
  23. 23. Image and Video de-noising: • Wiener Filter Filtered
  24. 24. Image and Video de-noising: • Non-Local Means Filter • Each pixel is replaced by weighted average of its neighborhoods. • Larger patch size leads to expensive computation. while Smaller patch size results in poor de-noising effect. • All weights should be normalized .
  25. 25. Image and Video de-noising: • Non-Local Means Filter σ n = 15
  26. 26. Image and Video de-noising: • Non-Local Means Filter
  27. 27. Image and Video de-noising: • Comparison original Wiener filter Noisy σ n = 15 NLM filter
  28. 28. Image and Video de-noising: • Comparison PSNR (db) 14 12 10 8 6 4 2 0 PSNR (db)
  29. 29. Video de-noising: • Each pixel is replaced by 3D spatio-temporal weighted average of its neighborhoods.
  30. 30. Motion estimation & Kalman Filter • Optical Flow Algorithms • Block Matching Algorithms • Full Search Algorithm (FSA) • Three Step Algorithm (TSA) • Four Step Algorithm (FSA) • Diamond Search Algorithm (DSA) • Adaptive Rood Pattern Search
  31. 31. Motion estimation & Kalman Filter • Why Kalman Filter? • Motion estimation inaccuracy when illumination is changed • Inaccuracy based on noise • The size of moving object may differ from searching region size • Fails in presence of occlusion
  32. 32. Motion estimation & Kalman Filter 2D Causal AR model: B(m,n-1) B(m-1,n) B(m,n) vi , x (m, n) = a01vi , x (m, n − 1) + a10vi , x (m − 1, n) + wi , x (m, n) vi , y (m, n) = b01vi , y (m, n − 1) + b10vi , y (m − 1, n) + wi , y (m, n)
  33. 33. Motion estimation & Kalman Filter State space representation :  vi , x (m, n)  a01 a11 0 0  vi , x (m − 1, n) q    v (m − 1, n)  1 0 0 0  vi , x (m, n − 1) 0  i,x =  * +   vi , x (m, n)   0 0 b01 b11  vi , x (m − 1, n) q          vi , x (m − 1, n)  0 0 1 0  vi , x (m, n − 1) 0    vi , x (m, n)   r    vi , x (m, n)  1 0 0 0 vi , x (m − 1, n) 0 +  = z= * vi , y (m, n) 0 0 1 0  vi , x (m, n)   r         vi , x (m − 1, n) 0  q 2  0 Q= 2 q  0  0 q2 0 0 0 q2 r 2  0 R= 2 r  0  0 r2 0 0 0 r2 0 0 0 0 0  0 0  0  0  0 0  0 
  34. 34. Motion estimation & Kalman Filter
  35. 35. Motion estimation & Kalman Filter
  36. 36. Motion estimation & Kalman Filter
  37. 37. Motion estimation & Kalman Filter
  38. 38. Motion estimation & Kalman Filter
  39. 39. Motion estimation & Kalman Filter • Green arrows show motion vectors • Red arrows show corrected motion vectors • Kalam uses corrected version of neighboring vectors if available
  40. 40. Motion estimation Results [Refer to Video2] • Results are generated using the following approaches: • 2D 3x3 spatial domain NLM • 3D 3x3x3 spatio-temporal domain NLM • 3D 5x3x3 spatio-temporal domain NLM
  41. 41. Motion estimation Results [Refer to Video2]
  42. 42. Kalman Tracking for Robust pose estimation • Object Recognition • Object recognition requires set of 2D correspondences between target image and video frame. • Robust RANSAC (RANdom SAmple Consensus) is used to robustly estimate the pose against wrong matches. • Robust pose estimation brings an exhaustive computational cost.
  43. 43. Kalman Tracking for Robust pose estimation • Tracking • Feature points tracking • Tracking the region of search for matches. • Keep tracking of estimated pose after pose estimation process. • Keep tracking of inliers (best matches) found from RANSAC loop. (My Contribution)
  44. 44. Kalman Tracking for Robust pose estimation • Pose estimation: Video Frame points Target Points Point Matching RANSAC Loop M-estimator (LSM) Kalman tracker
  45. 45. Kalman Tracking for Robust pose estimation • Pose estimation: [Refer to Video3]
  46. 46. Implementation • Programs are written with C++ using OpenCV library • C++ implementation of generic Kalman Filter algorithm • C++ implementation of Non-Local Means algorithm • C++ implementation of FSA Block Matching algorithm • C++ implementation of Wiener filter for image denoising • Matlab full implementation of Conventional, Extended , Adaptive Fading and Innovation-based Adaptive Kalman filter Will be available for download soon on https://github.com/hamidb
  47. 47. Conclusion: • We studied Kalman Filter as an adaptive filter which can be applied for Smoothing, prediction , filtering and tracking • Three main extensions of Kalman filter including conventional, extended and adaptive Kalman filter are reviewed. • Applying Kalman Filter in Video de-noising and compared the result with other filters like Wiener filter • Applying Kalman filter in computer vision applications such as multiple view geometry and pose estimation
  48. 48. References: 1. Dan Simon. Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches. Wiley-Interscience, 1 edition, August 2006. 2. S. Sangsuk-Iam and T. E. Bullock. Analysis of discrete time kalman filtering under incorrect noise covariances. Automatic Control, IEEE Transactions 3. Qijun Xia, Ming Rao, Yiqun Ying, and Xuemin Shen. Adaptive fading kalman filter with an application. Automatica 4. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2005) 5. Buades, A., Coll, B., Morel, J.M.: Denoising image sequences does not require motion estimation. In: IEEE International Conference on Advanced Video and Signal Based Surveillance. (2005) 6. Han, Yu Bing and Chen, Rushan: Efficient video denoising based on dynamic nonlocal means . Journal of Image Vision Comput. (2012) 7. Chung-Ming Kuo; Chaur-Heh Hsieh; Yue-Dar Jou; Hsieh-Cheng Lin; Po-Chiang Lu, "Motion estimation for video compression using Kalman filtering," Broadcasting, IEEE Transactions on , Jun 1996 8. Bazargani, H.; Omidi, E.; Talebi, H.A., "Adaptive Extended Kalman Filter for asynchronous shuttering error of stereo vision localization," Robotics and Biomimetics (ROBIO), 2012 IEEE International Conference on , vol., no., pp.2254,2259, 11-14 Dec. 2012 9. Ruiz, V.; Fotopoulos, V.; Skodras, A.N.; Constantinides, A.G., "An 8×8-block based motion estimation using Kalman filter," Image Processing, 1997. 10. Antoni Buades and Bartomeu Coll and Jean Michel Morel, On image denoising methods,Technical Note, CMLA (Centre de Mathematiques et de Leurs Applications,(2004)
  49. 49. Thank you for your Attention. http://www.slideshare.net/hamidbazargani/venndiagrams-29181371
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×