Coupling Eye-Motion and Ego-Motion Features forFirst-Person Activity Recognition
•Download as PPTX, PDF•
0 likes•749 views
Slide for CVPR workshop
![First-Person Activity Recognition
• Recognize tasks from [Sun 09]
wearable sensors
– Cameras, accelerometers, an
d IMUs
[Fathi 11]
– Previous: kinematic motions
• Changes in attention could
provide another source of
information
2](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Coupling Eye-Motion and Ego-Motion Features forFirst-Person Activity Recognition
- 1. Coupling Eye-Motion and Ego-Motion Features for First-Person Activity Recognition
- 2. First-Person Activity Recognition • Recognize tasks from [Sun 09] wearable sensors – Cameras, accelerometers, an d IMUs [Fathi 11] – Previous: kinematic motions • Changes in attention could provide another source of information 2
- 3. Previous Work Activity recognition using changes in attention 1.Eye-motion First person activity recognition using eye-motion [Bulling 11] – Independent use of eye- motion cannot represent broader changes of attention. Reading a book electrooculography 3
- 4. Previous Work Activity recognition using changes in attention 1.Eye-motion 2.Ego-motion Reading a book Copying text from one display to another 4
- 5. Eye-motion Characteristics [Bulling 11] Gaze-x • Micro level Intermittent jumps “Saccade” • Macro level Activities involves characteristic saccade sequences – Histogram of motion sequence “Wordbook” Time 6
- 6. Eye-motion and Ego-motion Eye-motion Right • Reading a book Left Right Ego-motion • Copying text from one display to another Left Both of two motions have Intermittent motion primitives and characteristic sequences 7
- 7. Our Approach Our method: Inside-out camera system provide eye-motion and ego-motion at once. Eye-motion Ego-motion 8
- 8. Overview Inside Outside Gaze coordinates Frames Motion Extraction Eye-motion Ego-motion Wordbook Analysis Wordbook Feature Classification 12
- 9. Overview Inside Outside Gaze coordinates Frames Motion Saccade Extraction detection Eye-motion Ego-motion Wordbook Analysis Wordbook Feature Classification 12
- 10. Eye-motion Primitives [Bulling 11] Inside Gaze coordinates Saccade detection Eye-motion Primitive Time Threshold & Quantization Eye-motion Characters NOUA B K r b Mn u b L l r C R K j d f E Time J HDG F 11
- 11. Eye-motion Primitives [Bulling 11] Inside Gaze coordinates Saccade detection Eye-motion Primitive (Saccade) Time Threshold & Quantization Eye-motion Characters NOUA B K r b Mn u bb C L l r r R KK j d f J HDG E F Time 11
- 12. Overview Inside Outside Gaze coordinates Frames Motion Saccade Global Extraction detection Optical flow Eye-motion Ego-motion Wordbook Analysis Wordbook Feature Classification 12
- 13. Ego-motion Primitives Outside Frames Global optical flow Ego-motion Primitive Time Threshold & Quantization Ego-motion Characters NOUA B Mn u b C U r b r A L l r K j d f R E Time J HDG F 11
- 14. Overview Inside Outside Gaze coordinates Frames Motion Saccade Global Extraction detection Optical flow Eye-motion Ego-motion Wordbook n-gram Wordbooks Analysis Wordbook Feature Classification 12
- 15. Generating Wordbook [Bulling 11] Motion primitives … U r b r A … Time Wordbook features … Current Frame Retrieve 1-4 gram wordbooks 1-gram Word count r 100 L 50 … H 1 16
- 16. Generating Wordbook [Bulling 11] Motion primitives … U r b r A … Time Wordbook features … Current Frame Retrieve 1-4 gram wordbooks 1-gram 4-gram Word count Word count r 100 rrrr 5 L 50 … Lrrr 2 … … H 1 lrlr 1 17
- 17. Generating Wordbook [Bulling 11] Motion primitives … U r b r A … Time Wordbook features … Current Frame Retrieve 1-4 gram wordbooks 1-gram 4-gram Statistical feature Word count Word count max r 100 rrrr 5 L 50 … Lrrr 2 size … … H 1 lrlr 1 range, var, mean 18
- 18. Generating Wordbook [Bulling 11] Motion primitives … U r b r A … Time Wordbook features … Current Frame Retrieve 1-4 gram wordbooks 1-gram 4-gram Statistical feature Word count Word count max r 99 rrrr 5 L 50 … Lrrr 2 size … … H 1 lrlr 1 range, var, mean 19
- 19. Generating Wordbook [Bulling 11] Motion primitives … U r b r A … Time Wordbook features … Current Frame Retrieve 1-4 gram wordbooks 1-gram 4-gram Statistical feature Word count Word count max r 100 rrrr 5 L 49 … Lrrr 3 size … … H 1 lrlr 1 range, var, mean 20
- 20. Overview Inside Outside Gaze coordinates Frames Motion Saccade Global Extraction detection Optical flow Eye-motion Ego-motion Wordbook n-gram Wordbooks Analysis Wordbook Feature 2 types, 4 wordbooks, 5 features Classification =40dim feature Linear SVM 12
- 21. Experiment 1.READ 2.VIDEO 3.WRITE 4.COPY 5.BROWSE 1 2 3 4 5 1 2 3 4 5 30 sec. 120 sec. 25 min. / subject Training Test • 5 subjects Case 1 Workspace 1 Workspace 2 • 2 sets / subject Case 2 Workspace 2 Workspace 1 22
- 22. Comparing the Feature Performance Saccade Wordbook (SW) Eye-motion feature [1] Motion Histogram (MH) Ego-motion feature [2] Motion Wordbook (MW) Ego-motion feature (proposed) GIST Visual feature [3] [1] Bulling, 11 [2] Kitani, 11 [3] Oliva, 01 19
- 23. Performance Over Tasks • MOWORD feature improves the performance • Joint feature improves the performance complementaly 20
- 24. Summary Joint use of eye-motion and ego-motion improves classification performance Future Work • Apply to a wider range of tasks Ego-motion feature might do more on dynamic tasks • Combined of object detection Hand, face, or other objects