A real time automatic eye tracking system for ophthalmology

769
-1

Published on

Presentation of my senior Project about "A real time automatic eye tracking system for ophthalmology"

In the presentation, it briefly explains about conventional object tracking method "template matching" based on Sum-of-Square difference. Therefore we also present the powerful matching technique called Gradient Orientation Pattern Matching (GOPM) proposed by T.Kondo and we proposed an improved version of GOPM called time-vary GOPM to solve a illumination and noise problem.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
769
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
22
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A real time automatic eye tracking system for ophthalmology

  1. 1. A Real-Time Automatic Eye Tracking System for Ophthalmology Mr. Wattanit Hotrakool Mr. Prarinya Siritanawan Supervised by Dr.Toshiaki Kondo Sirindhorn International Institute of Technology1
  2. 2. Outline Project Background Project Objective A eye-tracking technique using Traditional Template Matching A eye-tracking technique using Gradient Orientation Pattern Matching A eye-tracking technique using Time-varying GOPM Conclusion Question and Answer 2
  3. 3. Project Background The conventional eye-surgery cameras are manual- controlled and they reduce the efficiency of surgery. In order to reduce the burden of oculist, the automated camera control is required. The image processing is used to locate and track the eye’s centroid. 3
  4. 4. Project Background Many real-time eye tracking techniques used intensity data as an input; they are very sensitive to changing lighting condition and result as miss-matching. This project proposes new template matching based technique which robust to changing lighting condition by using Time-varying Gradient Orientation Pattern Matching. 4
  5. 5. Project Objectives To implement matching-based techniques in real-time. To verify the robustness to changing lighting condition of gradient orientation pattern matching. To develop new eye-tracking technique; a time-varying gradient orientation pattern matching. 5
  6. 6. Eye-tracking method using templatematching technique Template matching is the intensity-based technique for measuring the similarity between template and corresponding block of image. Template Match Sample frame 6
  7. 7. Simulation Specification Simulation System using the following hardware and software specifications: Software Specification Operating System Windows/Linux Programming Language C/C++ Primary library OpenCV 2.0 Hardware Specification Processor Intel Core2 Duo Processor Speed 1.66 GHz Memory 4GB 7
  8. 8. Video sequences used for simulation Video sequences used in this simulation can be categorized into 3 categories:1. Test video in normal lighting condition2. Test video in changing lighting condition3. Actual surgery video from real camera 8
  9. 9. Template Initialization Template initialization is required before the eye-tracking method. The pupil is specified as a template in order to track the eye. The example of video and corresponding template for each categories is shown in next page 9
  10. 10. Test video in normal lighting condition Video sequence Template10
  11. 11. Test video in changing lighting condition Video sequence Template11
  12. 12. Actual surgery video from real camera Video sequence Template12
  13. 13. A eye-tracking method using Traditional template matching13
  14. 14. A eye-tracking method using Traditionaltemplate matching There are many traditional techniques of template matching such as sum-of-absolute difference (SAD), sum- of-squared difference (SSD), or cross-correlation (CC) technique. In this step, we implement the sum-of-squared difference technique (SSD) to be eye-tracking method. 14
  15. 15. Sum-of-squared Difference technique SSD is the template matching method done by finding the lowest difference value between input and template. The differences are squared in order to remove the sign. where I1 is the intensity of input image and I2 is the intensity of template N is the size of the template 15
  16. 16. Procedure1. Convert template to gray scale image.2. Convert input frame to gray scale image.3. For every pixel, compute the SSD between input and template.4. Find the minimum difference pixel, which is the best matching location.5. For every frame, repeat step 2-4. 16
  17. 17. Procedure17
  18. 18. Result Input Average computation time Precision error (ms) (%) Normal light video 78.35 1.33(resolution: 320x240 px) Changing light video 78.63 40.47(resolution: 320x240 px) Actual surgery video 103.33 0(resolution: 384x288 px) Average computation time mostly depends on video resolution However, this method currently can process at 10 -12.5 frame/sec18
  19. 19. Result Input Average computation time Precision error (ms) (%) Normal light video 78.35 1.33 (resolution: 320x240) Changing light video 78.63 40.47 (resolution: 320x240) Actual surgery video 103.33 0 (resolution: 384x288) SSD technique can work very well in normal light video. However, this technique give high error in changing light video because it uses intensity data which are sensitive to light. Therefore SSD cannot work in changing light condition19
  20. 20. Result Properties of eye-tracking using SSD technique Obstacle robustness Yes Blur robustness Yes Light condition robustness No Scaling robustness No Average computation time About 50-350 ms20
  21. 21. A eye-tracking method using Gradient Orientation Pattern Matching  Presented at ICESIT2010, Chiang Mai, Thailand21
  22. 22. A eye-tracking technique using GradientOrientation Pattern Matching In order to develop a method that can provide the robustness to light condition, the new template matching technique is used. The gradient orientation pattern matching (GOPM) is a new template matching technique proposed by Dr. Toshiaki Kondo. 22
  23. 23. Gradient Orientation Pattern Matching GOPM is a template matching method which use the normalized gradient of the image in place of intensity data. Thus, it only consider about the shape of the pattern but not light. Gradient vector is the first derivative of intensity. The gradient in x and y direction are defined as: where I is a intensity of an image 23
  24. 24. Gradient Orientation Pattern Matching The gradient in x and y will then be normalized. This step provides the robustness to light condition. The normalized gradient in x and y direction are defined as: where And is a small constant used to prevent zero-division. 24
  25. 25. Gradient Orientation Pattern Matching The normalized gradient in x and y direction of input frame and template will be match by using SSD where N1 is the normalized gradient of input image and N2 is the normalized gradient of template 25
  26. 26. Procedure For every frame, we can divide the procedure into 2 main steps;1. Gradient orientation information (GOI) extraction2. Gradient orientation pattern matching (GOPM) 26
  27. 27. Gradient Orientation Information (GOI)Extraction Extract the gradient Images (Template and sample frame) in x and y direction.27
  28. 28. Gradient Orientation Pattern Matching Apply template matching in x and y direction. Then add the result of x and y direction28
  29. 29. Result Input SSD technique GOPM technique Average Precision Average Precision computation error (%) computation error (%) time (ms) time (ms)Normal lighting condition 78.35 1.33 62.48 0(resolution: 320x240px)Change lighting condition 78.63 40.47 63.09 12.87(resolution: 320x240px) Actual surgery camera 103.33 0 77.7 0(resolution: 384x288px) Average computation time of GOPM slightly inprove from SSD Even though the method is more complex, but the computation time is decrease due to the variable type and internal structure of OpenCV However, this method still can process at only <15 frame/sec 29
  30. 30. Result Input SSD technique GOPM technique Average Precision Average Precision computation error (%) computation error (%) time (ms) time (ms)Normal lighting condition 78.35 1.33 62.48 0(resolution: 320x240px)Change lighting condition 78.63 40.47 63.09 12.87(resolution: 320x240px) Actual surgery camera 103.33 0 77.7 0(resolution: 384x288px) In changing light condition, GOPM error is dramatically decrease due to the normalized process. Therefore GOPM can provide the robustness to changing light condition 30
  31. 31. Result Properties of eye-tracking using SSD and GOPM technique SSD GOPM Obstacle robustness Yes Yes Blur robustness Yes Yes Light condition robustness No Yes Scaling robustness No No Average computation time About 50-350 ms About 50-350 ms31
  32. 32. A eye-tracking method using a Time-varying GOPM  Accepted by ECTI-CON 2010, Chiang Mai, Thailand32
  33. 33. Time-varying GOPM Even though GOPM provides robustness to changing light condition, however the static template will not guaranteed that it yields the good result for all condition. There are many uncontrolled factors such as skin and noise. Time-varying GOPM uses the dynamic template which update itself automatically in place of static template. It reduce the difference of template environment in various period of time. 33
  34. 34. Template Update Algorithm Step 1 : Perform GOPM, get best matching coordinate current Template BEST MATCH Sample frame 34
  35. 35. Template Update Algorithm Step 2 : Crop region with the same size of old template for creating new template new Template Sample frame 35
  36. 36. Correct-matching criterion 1st Criterion Equation where Nxn+1 and Nyn+1 are the normalized gradient of the newly created template, Nxn and Nyn are the normalized gradient of the current template, i and j are the size of the template. is a threshold value defined as  1st Criterion is used to check the correctness of the updated template and prevent the jumping coordinate. 36
  37. 37. Correct-matching criterion 2nd Criterion Equation where Xn and Yn are the location of current matching result, Xc and Yc are the location of the last known correct result. T is a threshold value define as  2nd Criterion is used to double check the jumping real coordinate. 37
  38. 38. 1st Criterion STEP 1 : Find the difference b/w gradient component of old template and new template in X and Y directionx - =y - = Old Template New Template Template diff. 38
  39. 39. 1st Criterion STEP 2 : Combine the difference of x and y + = Diff x Diff y Total Diff STEP 3 : Sum all elements and then thresholding  If summation is less than the threshold function, update template.  If summation is more than the threshold function, discard the new template. 39
  40. 40. 312nd Criterion Using the fact that It is impossible that the eye would change its position suddenly in next frame. (400,100) (300,500) Frame N Frame N+140
  41. 41. 2nd Criterion STEP 1 : Find the best matching of the frame N. (x1,y1) Frame N 41
  42. 42. 2nd Criterion STEP 2 : If location of N passed the criteria, the location is used as a latest known correct position C. Frame N 42
  43. 43. 2nd Criterion STEP 3 : Find best matching of the frame N+1. (x2,y2) Frame N+1 43
  44. 44. 2nd Criterion STEP 4 : Find Euclidean distance between position C and position in frame N+1. Frame N+1 44
  45. 45. 2nd Criterion  STEP 5.1 : If distance more than threshold function, discard the current location.Threshold fcn Frame N+1 45
  46. 46. 2nd Criterion  STEP 5.2 : If distance less than threshold function, mask the location as new corrected position C.Threshold fcn Frame N+1 46
  47. 47. Correct-matching criteria with TemplateUpdate47
  48. 48. Procedure48
  49. 49. Downsampling In here, we resizes input video sequence and template to 50 percent of the height and width. Hence the downsampled image is reduced to ¼ of the original size. Thus, computation time is 4 times faster. No effect to the matching result since both video and template are downsampled with the same ratio 49
  50. 50. Result Input GOPM technique Time-varying GOPM Average Precision Average Precision computation error (%) computation error (%) time (ms) time (ms)Normal lighting condition 62.48 0 13.81 0(resolution: 320x240 px)Change lighting condition 63.09 12.87 12.92 0(resolution: 320x240 px) Actual surgery camera 77.7 0 17.12 0(resolution: 384x288 px) Average computation time is decreased by downsampling Currently this method can process at > 50 frame/sec which is enough for most of video capture device that run at 25 frame/secs. 50
  51. 51. Result Input GOPM technique Time-varying GOPM Average Precision Average Precision computation error (%) computation error (%) time (ms) time (ms)Normal lighting condition 62.48 0 13.81 0(resolution: 320x240 px)Change lighting condition 63.09 12.87 12.92 0(resolution: 320x240 px) Actual surgery camera 77.7 0 17.12 0(resolution: 384x288 px) In all cases, time-varying GOPM provide the better result than normal GOPM. Especially for the case of changing light condition, which error is decrease to 0% Therefore time-varying GOPM can provide the robustness to changing light condition with more precision than normal GOPM 51
  52. 52. Result Properties of eye-tracking using SSD, GOPM , and time-varying GOPM technique SSD GOPM Time-varying Obstacle robustness Yes Yes Yes Blur robustness Yes Yes Yes Lighting condition robustness No Yes Yes Scaling robustness No No No Average computation time About 50-350 About 50-350 About 10-90 ms ms ms52
  53. 53. Drawback of time-varying GOPM In rarely case, when the 2nd criteria drop the frame repeatedly, it causes the template slightly shifts from the eye’s centroid. Fail update However for real implementation prospective, if not be too much, can be tolerated by the surgeon 53
  54. 54. Conclusion54
  55. 55. Conclusion This work verify that the speed of template matching technique with downsampling is able to implement in real-time. (speed > 50 frame/secs). In the changing light condition, the result clearly shows that GOPM is more robust than SSD. A time-varying GOPM reduce the difference of template environment in various time and provides the higher precision of tracking than normal GOPM. 55
  56. 56. Future Work Optimize the utilization of the threshold function in corrected-matching criterion. Due to difference in camera specification such as resolution or sensitivity, it required other advance method to supervised the threshold function such as machine learning or neuron network. 56
  57. 57. Acknowledgement Assist. Prof. Dr. Toshiaki Kondo Assoc.Prof. Dr. Waree Kongprawechnon Dr. Itthisek Nilkhamhang All faculty members and our beloved friends 57
  58. 58. Question and Answer58

×