Rule-Based High-Level Situation Recognition           from Incomplete Tracking Data                    David Münch1, Joris...
MotivationMotivation•     Security systems: Persons      and their behavior are      the focus of attention:      Person c...
Overview• Cognitive Vision System as a whole.• High-level knowledge and situation recognition.• Handling Incomplete Data.•...
Cognitive Vision System                          4© Fraunhofer IOSB
Cognitive Vision System                          5© Fraunhofer IOSB
Cognitive Vision System                          6© Fraunhofer IOSB
Cognitive Vision System                          7© Fraunhofer IOSB
Cognitive Vision System                          8© Fraunhofer IOSB
Cognitive Vision System                          9© Fraunhofer IOSB
Conceptual Layer – Conceptual Primitives Level•     Quantitative information (from Quantitative Layer) is transformed into...
Dictionary of basic rules   Dictionary of basic rules for every   domain.   Fuzzy Metric Temporal Logic   (FMTL):   Extens...
“Numbers” mapped to Concepts  Fuzzy membership functions 𝜇 𝑠𝑝𝑒𝑒𝑑 𝑣𝑎𝑙𝑢𝑒 for the subset {zero, small, normal,  high, very_hi...
Conceptual Layer – Behavior Representation LevelHow to represent the expected Situations?•     Knowledge represented in Si...
Behavior Representation LevelBasic logic predicates from Conceptual Primitives Level are aggregated andstructured in Situa...
Behavior Representation Level                                15© Fraunhofer IOSB
Handling Incomplete Data – Low Level in Scene Domain Level           Perfect input data.                                  ...
Handling Incomplete Data – Low Level in SDL           Incomplete input data.                                              ...
Handling Incomplete Data – Low Level in SDLInterpolation of input data:                                              Truth...
Handling Incomplete Data – Low Level in SDLInterpolation of input data:                                              Time ...
Handling Incomplete Data – High Level in BRL                               20© Fraunhofer IOSB
Handling Incomplete Data – High Level in BRL                    What if is_open(trunk, Car) fails?                        ...
Handling Incomplete Data – High Level in BRLHallucination (abduction) of missing evidence.                    What if is_o...
VIRAT Video Dataset  VIRAT Video Dataset Release 1.0  Input data: annotated ground truth: persons and their situations.  P...
Situations1. Person loads object into car.2. Person unloads object from car.3. Person gets into car.4. Person gets out of ...
Evaluation•     The annotated ground truth is regarded as complete information.•     Randomly make gaps of a distinct leng...
Evaluation  Original,              F-Score  unmodified                         Gap size                         5 seconds....
Evaluation  Original,              F-Score  unmodified                         Gap size                         5 seconds....
Evaluation      ROC-curves for video (d) with gap size 5 seconds.   TPR                                       TPR         ...
Evaluation  Video (d),  F-Score                Gap size                         5 seconds.Gap size                 Gap siz...
Conclusion•     low level: interpolation of data and its uncertainty.      ordinary incomplete data.•     high level: ext...
Finis.                      31© Fraunhofer IOSB
Upcoming SlideShare
Loading in …5
×

Ruleml2012 - Rule-based high-level situation recognition from incomplete tracking data

644 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
644
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Ruleml2012 - Rule-based high-level situation recognition from incomplete tracking data

  1. 1. Rule-Based High-Level Situation Recognition from Incomplete Tracking Data David Münch1, Joris IJsselmuiden1, Ann-Kristin Grosselfinger1, Michael Arens1, and Rainer Stiefelhagen1,2 1 Fraunhofer IOSB, Germany, david.muench@iosb.fraunhofer.de 2 Karlsruhe Institute of Technology, Germany. The 6th International Symposium on Rules - Rule ML2012, Montpellier, France, August 27-29, 2012 1© Fraunhofer IOSB
  2. 2. MotivationMotivation• Security systems: Persons and their behavior are the focus of attention: Person centric analysis• Threat detection• Visual surveillance• Activity logging• Video search• Driver Assistance Systems Input data is incomplete and noisy. 2© Fraunhofer IOSB
  3. 3. Overview• Cognitive Vision System as a whole.• High-level knowledge and situation recognition.• Handling Incomplete Data.• Experiments. 3© Fraunhofer IOSB
  4. 4. Cognitive Vision System 4© Fraunhofer IOSB
  5. 5. Cognitive Vision System 5© Fraunhofer IOSB
  6. 6. Cognitive Vision System 6© Fraunhofer IOSB
  7. 7. Cognitive Vision System 7© Fraunhofer IOSB
  8. 8. Cognitive Vision System 8© Fraunhofer IOSB
  9. 9. Cognitive Vision System 9© Fraunhofer IOSB
  10. 10. Conceptual Layer – Conceptual Primitives Level• Quantitative information (from Quantitative Layer) is transformed into primitive conceptual knowledge (Logic predicates).• Dictionary of basic rules.• Mainly domain independent.• Support of uncertainty and vagueness.• The rules in the CPL are mostly concerned with spatial relations and temporal relations on short time intervals. 10© Fraunhofer IOSB
  11. 11. Dictionary of basic rules Dictionary of basic rules for every domain. Fuzzy Metric Temporal Logic (FMTL): Extension of first order logic by notions of fuzziness, time, and metrics on time. Inference engine: F-LIMETTE 11© Fraunhofer IOSB
  12. 12. “Numbers” mapped to Concepts Fuzzy membership functions 𝜇 𝑠𝑝𝑒𝑒𝑑 𝑣𝑎𝑙𝑢𝑒 for the subset {zero, small, normal, high, very_high} of discrete conceptual speed values. Figure from: H.-H. Nagel, Steps toward a Cognitive Vision System, 2004, AI Mag. 12© Fraunhofer IOSB
  13. 13. Conceptual Layer – Behavior Representation LevelHow to represent the expected Situations?• Knowledge represented in Situation Graph Trees. • Graphically editable • Easy to extend and edit • Interpretable (vs. black box)• Exhaustive situation recognition. 13© Fraunhofer IOSB
  14. 14. Behavior Representation LevelBasic logic predicates from Conceptual Primitives Level are aggregated andstructured in Situation Graph Trees (SGT)  high-level conceptual situations.An SGT consists of situation schemes:• Can be start and/or end nodes.• Unique name.• State scheme (Precondition).• Action scheme (Postcondition).Specialize a situation scheme:• Prediction edges link to a possible subsequent situation scheme.• Specialization edges link to more specific situation graphs in a hierarchical structure. 14© Fraunhofer IOSB
  15. 15. Behavior Representation Level 15© Fraunhofer IOSB
  16. 16. Handling Incomplete Data – Low Level in Scene Domain Level Perfect input data. Truth value Time 16© Fraunhofer IOSB
  17. 17. Handling Incomplete Data – Low Level in SDL Incomplete input data. Truth value Time 17© Fraunhofer IOSB
  18. 18. Handling Incomplete Data – Low Level in SDLInterpolation of input data: Truth value Time 18© Fraunhofer IOSB
  19. 19. Handling Incomplete Data – Low Level in SDLInterpolation of input data: Time 19© Fraunhofer IOSB
  20. 20. Handling Incomplete Data – High Level in BRL 20© Fraunhofer IOSB
  21. 21. Handling Incomplete Data – High Level in BRL What if is_open(trunk, Car) fails? 21© Fraunhofer IOSB
  22. 22. Handling Incomplete Data – High Level in BRLHallucination (abduction) of missing evidence. What if is_open(trunk, Car) fails?  Hallucinate is_open(trunk, Car) and continue! 22© Fraunhofer IOSB
  23. 23. VIRAT Video Dataset VIRAT Video Dataset Release 1.0 Input data: annotated ground truth: persons and their situations. Place: 0000: Place: 0002: Videos: 02 03 04 06 Videos: 00 06 23© Fraunhofer IOSB
  24. 24. Situations1. Person loads object into car.2. Person unloads object from car.3. Person gets into car.4. Person gets out of car. VIRAT Video Dataset Release 1.0 24© Fraunhofer IOSB
  25. 25. Evaluation• The annotated ground truth is regarded as complete information.• Randomly make gaps of a distinct length into the data.• Increase the amount of missing data in steps of 5%.• Repeat each experiment several times. 25© Fraunhofer IOSB
  26. 26. Evaluation Original, F-Score unmodified Gap size 5 seconds. Precision RecallGap size Gap size5 seconds. 5 seconds. 26© Fraunhofer IOSB
  27. 27. Evaluation Original, F-Score unmodified Gap size 5 seconds. Precision RecallGap size Gap size5 seconds. 5 seconds. 27© Fraunhofer IOSB
  28. 28. Evaluation ROC-curves for video (d) with gap size 5 seconds. TPR TPR FPR FPR With interpolation and hallucination. Without interpolation and hallucination. 28© Fraunhofer IOSB
  29. 29. Evaluation Video (d), F-Score Gap size 5 seconds.Gap size Gap size5 seconds. 5 seconds. 29© Fraunhofer IOSB
  30. 30. Conclusion• low level: interpolation of data and its uncertainty.  ordinary incomplete data.• high level: extension of the situation recognition inference algorithm (hallucination, abduction).  high-level incomplete data such as occlusions.• Knowledge base for vehicle-centered situations.• Runs in real-time on off-the-shelf hardware. 30© Fraunhofer IOSB
  31. 31. Finis. 31© Fraunhofer IOSB

×