Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Eie426 eye tracking-v2 (1)

1,415 views

Published on

  • Be the first to comment

Eie426 eye tracking-v2 (1)

  1. 1. Eye Tracking Techniques and applications eie426-EyeTracking-200903.ppt03/06/13 EIE426-AICV 1
  2. 2. Outline Part I :Physiology of the EYE Part II : Mechanism of Eye Movements Part III :Eye Tracking Technology Part IV :Eye Tracking Applications03/06/13 EIE426-AICV 2
  3. 3. How does our eye work03/06/13 EIE426-AICV 3
  4. 4. Part I :Physiology of the EYE HOW DOES OUR EYE WORK03/06/13 EIE426-AICV 4
  5. 5. 03/06/13 EIE426-AICV 5
  6. 6. Part II : Mechanism of Eye Movements CAN YOU BELIEVE YOUR EYE03/06/13 EIE426-AICV 6
  7. 7. Part III :Eye Tracking Technology HOW EYE TRACKER WORKS03/06/13 EIE426-AICV 7
  8. 8. Outline1. Current State and Application of Eye Gaze Tracking (EGT) Technology2. Theory and the Classification of EGT Technology3. The Framework of EGT system
  9. 9. Ⅰ. Current State and Application of Eye Gaze Tracking (EGT) Technology
  10. 10. Ⅰ. Current State and Application of Eye Gaze Tracking (EGT) Technology What is EGT eye gaze——The line from the fovea through the center of the pupil is the line of sight (LoS). Usually, we take the optical axis as line of gaze (LoG). LoG can be approximate to LoS. In fact, LoS determines a person’s visual attention. eye gaze Tracking——By image Processing,if the LoG or LoS can be estimated, the point of regard (POR) is computed as the intersection of the LoG (or LoS) with the object of the scene or space. History and Development of EGT Interests of Visual attention can be traced back to 1897. At that time, it was a kind of diagnostic research, i.e. the recording of eye movement. Those technologies include ophthalmometer, Mechanical Method, Electro-Oculography (EOG), Optical-based Method , electromagnetic Oculography eye movement initially applied in medical research, such as brain and physiology analysis. with the development of Electronics, Computer and Image processing technology, further research focuses on Eye Gaze Tracking Technology .
  11. 11. Search Coil
  12. 12. A picture wearing the EOG apparatus
  13. 13. Ⅰ. Current State and Application of Eye Gaze Tracking (EGT) Technology eye tracking techniques : 1. Direct Observation : ophthalmometer, peep-hole method 2. Mechanic Method : use level to record eye movement 3. Optical-based Method : Reflection Method ( mirror or prism )、 Pupil- Corneal reflection 、 Purkinje Image 4. Electro-Oculography (EOG): recorded the difference in the skin potential 5. Electromagnetic Oculography : The users gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field Except Optical-based Method, the above methods are more or less low accuracy or high intrusiveness . As a result, Modern Eye Gaze Tracking Techniques are mostly based on Optical-based Method.
  14. 14. Ⅱ. Theory and the Classification of EGT Technology
  15. 15. Ⅱ. Theory and the Classification of EGT Technology  Image-Based EGT Technology 1. Infrared Light : out of visible light; Paired with filter to eliminate the light of other wavelength ; 2. Eye camera :tracking eye movement and recording eye image sequence 3. Image Processing: detecting the visual elements 4. Further Estimation: 。 Having pre-processed, the data of eye gaze can be get from eye-movement model
  16. 16. Ⅱ. Theory and the Classification of EGT Technology System FrameworkInput Image Image Feature acquisition Estimation Processing Output Data Eye Estimation Tracking Calibration
  17. 17. Ⅱ. Theory and the Classification of EGT TechnologyFunction of Each Module Image acquisition: 1 get bright pupil or dark pupil Image Processing : 2 Filtering ,Noise reducing ,difference ,thresholding Feature Estimation: 3 Estimating the center of Pupil and corneal reflection Eye Tracking: 4 Estimation of eye gaze Data Estimation: 5 Algorithm validation ,eye movement data analysis
  18. 18. Theory of EGT Technology  2D to detect the Optical Axis (basic)  2D to detect the Line of Sight (Advanced) ¹â Öá O A 2 D ·½ ·¨ ÊÓ Ïß Öá L O S 3 D ·½ ·¨
  19. 19. Classification of EGT Technology view of i n t r u s i v e £º C o n t a c t l e n s ¡¢ P h o t o - t r a n s m i t e r ¡¢ I R l i g h t e m i t t i n g d i o d e s humanity n o n - i n t r u s i v e £º u s e i m a g e - b a s e d M e t h o d D i a g n o s t i c £º U s e t h e E y e m o v e m e n t c h a r a c t e r s t o P s y c h o l o g y a n d N e u r o s c i e n c e view of ( n e e d to d is t in g u is h d if f e r e n t t y p e o f e y e m o v m e n t ) application I n t e r a c t i v e £º U s e E y e M o v m e n t t o d e c i d e P o i n t o f R e g a r d , u s e f o r i n t e r a c t i o n £¨ n e e d t o k n o w t h e v i s i u a l a t t e n t i o n £© B o d y - M o u n t e d £º g o g g l e ¡¢ h e l m e t ¡¢ b a c k p a c k , e t c n o t e : a l t h o u g h l a b e l e d w i t h i n r u s i v e ,b u t m o r e a n d m o r e b o d y - M o u n t e d s y s t e m s are b e c o m in g n o n -in tru s ive View of appearance R e m o t e s y s t e m £º R e m o t e E Y E G a z e T r a c k e r ( R E G T )
  20. 20. Ⅱ. Theory and the Classification of EGT TechnologyFrom View of Humanity Usually By goggle , helmet … Head features : high accuracy Mounted Eye Tracker defects : intrusiveness Usually on table Remote features : non-intrusive Eye Tracker defects : eye and head relative motion
  21. 21. Ⅱ. Theory and the Classification of EGT Technology From View of Theory The Purkinje images are reflections created at differentPurkinje Image layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possibleEOG to measure small differences in the skin potential corresponding to eye movement. The Limbus is the boundary between the white sclera and the darkLimbus-Scalar iris of the eye. By placing IR light emitting diodes and photo- transistors, respectively, above and below the eye. the resultingIR Tracking voltage difference is proportional to the angular deviation of the eye. Training images are taken when the user is lookingANN at a specific Calibration markers. Use ANN to decide the eye gazeContact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.Pupil-Corneal The IR source can generate a glint on corneal and divide pupil from iris, the difference between canreflection represent the eye gaze movement
  22. 22. Ⅱ. Theory and the Classification of EGT TechnologyFrom View of Theory ThePurkinje images are reflectionsreflections The Purkinje images are created at differentPurkinje Image layers of the eye structure. Eye gaze can be calculated created at different layers of the eye from these relative positions of these reflections structure. Eye gaze can be 将两对氯化银皮肤表面电极分别置于眼睛左右、上下EOG calculated from thesein the skin is possible By placing electrodes around the eye, it 两侧 , 就能引起眼球变化方向上的微弱电信号 , 经放 to measure small differences relativepotential positions ofto eye movement. corresponding these reflections 大后得到眼球运动的位置信息 The Limbus is the boundary between the white sclera and the darkLimbus-Scalar The Limbus is the boundary between the white iris of the eye. By placing IR light emitting diodes and photo- transistors, the dark iris of the eye. This eye. the resulting sclera andrespectively, above and below theboundary canIR Tracking voltage be opticallyproportional to the angular deviation of the eye. easily difference is detected and tracked Training images are taken when the user is lookingANN at a specific Calibration markers. Use ANN to decide the eye gazeContact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.Pupil-Corneal The IR source can generate a glint on corneal and divide pupil from iris, the difference between canreflection represent the eye gaze movement
  23. 23. Ⅱ. Theory and the Classification of EGT TechnologyFrom View of Theory The Purkinje images are reflections created at differentPurkinje Image layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possibleEOG to measure small differences in the skin potential corresponding to eye movement.Limbus-Scalar The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary canIR Tracking easily be optically detected and tracked Training images are taken when the user is lookingANN at a specific Calibration markers. Use ANN to decide the eye gazeContact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.Pupil-Corneal The IR source can generate a glint on corneal and divide pupil from iris, the difference between canreflection represent the eye gaze movement
  24. 24. Ⅱ. Theory and the Classification of EGT TechnologyFrom View of Theory The Purkinje images are reflections created at differentPurkinje Image layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possibleEOG to measure small differences in the skin potential corresponding to eye movement.Limbus-Scalar The Limbus isis the boundary betweenwhitewhite The Limbus the boundary between the the sclera and the dark iris of thethe eye. By placing IR sclera and the dark iris of eye. This boundary canIR Tracking light emitting diodes andand tracked easily be optically detected photo-transistors, respectively, above and below the eye. the 将两对氯化银皮肤表面电极分别置于眼睛左右、上下两 Training images are taken when the user is lookingANN resulting voltage difference is proportional to at a, specific 侧 就能引起眼球变化方向上的微弱电信号 , 经放大后 the angular Calibration of the eye. ANN to decide 得到眼球运动的位置信息 deviation markers. Use the eye gazeContact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.Pupil-Corneal The IR source can generate a glint on corneal and divide pupil from iris, the difference between canreflection represent the eye gaze movement
  25. 25. Ⅱ. Theory and the Classification of EGT TechnologyFrom View of Theory The Purkinje images are reflections created at differentPurkinje Image layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possibleEOG to measure small differences in the skin potential corresponding to eye movement.Limbus-Scalar The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary canIR Tracking easily be optically detected and tracked Training images are taken when the user is lookingANN at a specific Calibration markers. Use ANN to decide the eye gazeContact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.Pupil-Corneal The IR source can generate a glint on corneal and divide pupil from iris, the difference between canreflection represent the eye gaze movement
  26. 26. Ⅱ. Theory and the Classification of EGT TechnologyFrom View of Theory The Purkinje images are reflections created at differentPurkinje Image layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possibleEOG to measure small differences in the skin potential corresponding to eye movement.Limbus-Scalar The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary canIR Tracking easily be optically detected and tracked Use a small coil taken when the usera contact Training images are embedded into is lookingANN lens that is tightly fit over the sclera decide at a specific Calibration markers. Use ANN to . The user’s gaze is estimated from measuring the eye gaze theavoltageembedded into a contact lens that coil byfit induced in the search is tightly 将两对氯化银皮肤表面电极分别置于眼睛左右、上下Contact Lens Use small coil an the sclera . electro-magnetic field. , 经放 两侧 , 就能引起眼球变化方向上的微弱电信号 over externalThe user’s gaze is estimated from measuring the 大后得到眼球运动的位置信息 voltage induced in the search coil by an external electro-magnetic field.Pupil-Corneal The IR source can generate a glint on corneal and divide pupil from iris, the difference between canreflection represent the eye gaze movement
  27. 27. Ⅱ. Theory and the Classification of EGT TechnologyFrom View of Theory The Purkinje images are reflections created at differentPurkinje Image layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possibleEOG to measure small differences in the skin potential corresponding to eye movement.Limbus-Scalar The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary canIR Tracking easily be optically detected and tracked Training images are taken when the user is lookingANN at a specific Calibration markers. Use ANN to decide the eye gazeContact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.Pupil-Corneal The IR source can generate a glint on corneal and divide pupil from iris, the difference between canreflection represent the eye gaze movement
  28. 28. Ⅱ. Theory and the Classification of EGT Technology EGC Technology Accuracy Features High accuracy, but the light isPurkinje Image 0.017 0 hard to control and can only be used in Lab low robustness, low accuracy,EOG 1.50-2° high intrusiveness V 1 0 -7 0 Horizontal accuracy is betterLimbus Tracking than Vertical, but both are low H 0.5 0 -7 0 No need of calibration,lowANN 1.3-1.8° accuracy, high accuracy, highContact Lens 0.08 0 intrusivenessPupil Corneal Reflection 1° The best one till now
  29. 29. Ⅱ. Theory and the Classification of EGT Technology Feature of Eye Movement D ia g n o s t ic ( Õï ÁÆ Ê½ ) £º ÊÓ Ïß ×· ×Ù µÄ ½á ¹û ×÷ Ϊ Ò» ¸ö ¶¨ Á¿ ÌØ ÐÔ £¬ ¿¼ ²ì ÈË µÄ ÊÓ ¾õ ºÍ ÐÄ Àí ѧ ÑÐ ¾¿  view of  从系统 se C £¨ ¾ø ¶Ô ×¼ È· ÐÔ £¬ ¶Ô ÑÛ Çò ¶¯ ×÷ µÄ Çø ·Ö £© ine rt ar cu t iv iv (e½» £º»¥ ʽ o) £ºn tÖ»a cÒªt ×·le ×Ù s ³ö¡¢ ÊÓ P Ïßh £¬ t ×÷ - Ϊt r ºóa n ¶Ës m½» it»¥ e µÄ ¡¢Êä IÈëR lig h t e m it t in g d io d e s n o o r In t humanity 应用角度 n o n - in t r u s iv e £º£¨ Ïàu s¶Ôe ×¼imÈ· a £¬g eÒ»- °ãb a²» s Çø d ·Ö MÑÛ eÇòt hµÄo ÔË ¶¯ ÌØ ÐÔ £¬ Ö» Òª Öª µÀ ÊÓ Ïß ·½ Ïò £© e d T a x o n o m y a n d M o d e ls o f E y e M o v e m e n ts Saccades F ix a tio n a n g le : 1 o ~ 4 o S m o o t h P u r s u itd u r a t io n :1 0 0 ~ 2 0 0 m s d u r a t io n £º 3 0 ~ 1 2 0 m s v o le c it y £º 1 o / s ~ 3 0 o / s v o le c it y :4 0 0 o / s ~ 6 0 0 o / s
  30. 30. Ⅱ. Theory and the Classification of EGT Technology Development Early stage : Direct Observation ,Mechanical Method Initial Methods, not used any more for high intrusiveness and poor accuracy Development : EOG 、 Electromagnetic Oculography Although have improved a lot and widely used , these method are disappearing because of the intrusiveness Advanced : Optical Method : Because of its’ high accuracy and low intrusiveness, Optical Method have made rapid progress in recent years. 1. The Purkinje images are reflections created at different layers of the eye structure. and the eye gaze can be calculated from these relative positions of these reflections 2. Photo-Oculography : it measures the eye movement during its translation/rotation. with the IR light source, shape of the pupil ,Limbus or corneal reflection is detected 3. Limbus-Scalar IR Method: They place IR light emitting diodes and IR light sensitive photo- transistors, respectively, above and below the eye. Several such IR pairs can be mounted on goggles or helmets, a photo-transistor transforms the reflected IR light into a voltage. the resulting voltage difference is proportional to the angular deviation of the eye. 4. Pupil-Corneal Reflection: The IR source can generate a glint on corneal and divide pupil from iris, the camera can extract the pupil which represent the eye gaze, on the other hand the corneal reflection represent the head motion, as a result, the difference between them represents the real eye gaze movement. 。
  31. 31. Ⅱ. Theory and the Classification of EGT Technology Feature of Remote Eye Gaze Tracker B o d y - M o u n t e d £º g o g g l e ¡¢ h e l m e t ¡¢ b a c k p a c k , e t c n o t e : a l t h o u g h l a b e l e d w i t h i n r u s i v e ,b u t m o r e a n d m o r e b o d y - M o u n t e d s y s t e m s are b e c o m in g n o n -in tru s ive View of appearance Robustness R e m o t e s y s t e m £º R e m o t e E Y E G a ze T r a c k e r (R E G T )  Light Source  Eyelash, Eyelids and Camera Position  System Error  Eye station (dry or wet)  head motion Lower accuracy than Head Mounted Eye Tracker
  32. 32. Ⅲ A Practical Eye Gaze Tracking System
  33. 33. Ⅲ A Practical Eye Gaze Tracking System Framework of EGT System Hardware : a Pair of Cameras  eye camera  scene camera eye camera is for acquisition of pupil corneal reflection image, scene camera is for: 1 mapping from the eye image coordinate to scene image coordinate ;2 Showing the combining result of POR and scene in same image : M o r n ite r O u tp u t o f S c e n e V id e o S tr e a m V is u a l S t im u lu s E ye C a m era IR S o u rc e S cene C a m era T r a c k in g o f P u p il A u to -C o rn e a l M a p p in g R e fle c t io n O u tp u t o f E y e IR S o u rc e O p tic a l S y s te m V id e o S tr e a m P C
  34. 34. Ⅲ A Practical Eye Gaze Tracking Systemappearance of EGT System
  35. 35. Ⅲ Practical Eye Gaze Tracking System Theory of a Head Mounted Eye Tracker Capture Estimate Coordinate EyeEye Camera Pupil-Corne Pupil-Corneal Mapping Tracking al image reflection Result 22222222222 Calibration 222222222D Mapping between Eye xianglian Coordinate to Scene Tracking CoordinateScene Camera
  36. 36. Ⅲ Practical Eye Gaze Tracking SystemBright Pupil and Dark Pupil Bright Pupil Dark Pupil Difference Image When IR source is placed near the optical axis of the camera bright pupil as can be seen ;while IR is placed off the optical axis, a dark pupil can be seen. By Thresholding, a robust pupil contour can be extracted .
  37. 37. Ⅲ Practical Eye Gaze Tracking System Bright Pupil and Dark Pupil Bright Dark Pupil Pupil Corneal Reflection The IR source can generate a glint on corneal and divide pupil from iris, the camera can extract the pupil which represent the eye gaze, on the other hand the corneal reflection represent the head motion, as a result, the difference between them represents the real eye gaze movement.
  38. 38. Ⅲ Practical Eye Gaze Tracking System
  39. 39. Ⅲ Practical Eye Gaze Tracking System
  40. 40. Ⅲ Practical Eye Gaze Tracking System
  41. 41. Part IV :Applications HOW FAR YOU CAN FLY03/06/13 EIE426-AICV 41
  42. 42. Applications  Human Computer Interaction (HCI)—— efficiency, humanity  Intelligent Control —— EGT and Weapon Control  Human Movement Study ——Typing, physical training  Psychology —— Antinational Neuroscience,  Visual attention & Driving ——Aviation, navigation, driving, traffic accidents inspection  Scene and Image Perception—— Web, AD, Designing, Scene03/06/13 EIE426-AICV 42
  43. 43. Most Recent Application03/06/13 EIE426-AICV 52

×