Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ruleml2012 - Rule-based high-level situation recognition from incomplete tracking data

676 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Ruleml2012 - Rule-based high-level situation recognition from incomplete tracking data

  1. 1. Rule-Based High-Level Situation Recognition from Incomplete Tracking Data David Münch1, Joris IJsselmuiden1, Ann-Kristin Grosselfinger1, Michael Arens1, and Rainer Stiefelhagen1,2 1 Fraunhofer IOSB, Germany, david.muench@iosb.fraunhofer.de 2 Karlsruhe Institute of Technology, Germany. The 6th International Symposium on Rules - Rule ML2012, Montpellier, France, August 27-29, 2012 1© Fraunhofer IOSB
  2. 2. MotivationMotivation• Security systems: Persons and their behavior are the focus of attention: Person centric analysis• Threat detection• Visual surveillance• Activity logging• Video search• Driver Assistance Systems Input data is incomplete and noisy. 2© Fraunhofer IOSB
  3. 3. Overview• Cognitive Vision System as a whole.• High-level knowledge and situation recognition.• Handling Incomplete Data.• Experiments. 3© Fraunhofer IOSB
  4. 4. Cognitive Vision System 4© Fraunhofer IOSB
  5. 5. Cognitive Vision System 5© Fraunhofer IOSB
  6. 6. Cognitive Vision System 6© Fraunhofer IOSB
  7. 7. Cognitive Vision System 7© Fraunhofer IOSB
  8. 8. Cognitive Vision System 8© Fraunhofer IOSB
  9. 9. Cognitive Vision System 9© Fraunhofer IOSB
  10. 10. Conceptual Layer – Conceptual Primitives Level• Quantitative information (from Quantitative Layer) is transformed into primitive conceptual knowledge (Logic predicates).• Dictionary of basic rules.• Mainly domain independent.• Support of uncertainty and vagueness.• The rules in the CPL are mostly concerned with spatial relations and temporal relations on short time intervals. 10© Fraunhofer IOSB
  11. 11. Dictionary of basic rules Dictionary of basic rules for every domain. Fuzzy Metric Temporal Logic (FMTL): Extension of first order logic by notions of fuzziness, time, and metrics on time. Inference engine: F-LIMETTE 11© Fraunhofer IOSB
  12. 12. “Numbers” mapped to Concepts Fuzzy membership functions 𝜇 𝑠𝑝𝑒𝑒𝑑 𝑣𝑎𝑙𝑢𝑒 for the subset {zero, small, normal, high, very_high} of discrete conceptual speed values. Figure from: H.-H. Nagel, Steps toward a Cognitive Vision System, 2004, AI Mag. 12© Fraunhofer IOSB
  13. 13. Conceptual Layer – Behavior Representation LevelHow to represent the expected Situations?• Knowledge represented in Situation Graph Trees. • Graphically editable • Easy to extend and edit • Interpretable (vs. black box)• Exhaustive situation recognition. 13© Fraunhofer IOSB
  14. 14. Behavior Representation LevelBasic logic predicates from Conceptual Primitives Level are aggregated andstructured in Situation Graph Trees (SGT)  high-level conceptual situations.An SGT consists of situation schemes:• Can be start and/or end nodes.• Unique name.• State scheme (Precondition).• Action scheme (Postcondition).Specialize a situation scheme:• Prediction edges link to a possible subsequent situation scheme.• Specialization edges link to more specific situation graphs in a hierarchical structure. 14© Fraunhofer IOSB
  15. 15. Behavior Representation Level 15© Fraunhofer IOSB
  16. 16. Handling Incomplete Data – Low Level in Scene Domain Level Perfect input data. Truth value Time 16© Fraunhofer IOSB
  17. 17. Handling Incomplete Data – Low Level in SDL Incomplete input data. Truth value Time 17© Fraunhofer IOSB
  18. 18. Handling Incomplete Data – Low Level in SDLInterpolation of input data: Truth value Time 18© Fraunhofer IOSB
  19. 19. Handling Incomplete Data – Low Level in SDLInterpolation of input data: Time 19© Fraunhofer IOSB
  20. 20. Handling Incomplete Data – High Level in BRL 20© Fraunhofer IOSB
  21. 21. Handling Incomplete Data – High Level in BRL What if is_open(trunk, Car) fails? 21© Fraunhofer IOSB
  22. 22. Handling Incomplete Data – High Level in BRLHallucination (abduction) of missing evidence. What if is_open(trunk, Car) fails?  Hallucinate is_open(trunk, Car) and continue! 22© Fraunhofer IOSB
  23. 23. VIRAT Video Dataset VIRAT Video Dataset Release 1.0 Input data: annotated ground truth: persons and their situations. Place: 0000: Place: 0002: Videos: 02 03 04 06 Videos: 00 06 23© Fraunhofer IOSB
  24. 24. Situations1. Person loads object into car.2. Person unloads object from car.3. Person gets into car.4. Person gets out of car. VIRAT Video Dataset Release 1.0 24© Fraunhofer IOSB
  25. 25. Evaluation• The annotated ground truth is regarded as complete information.• Randomly make gaps of a distinct length into the data.• Increase the amount of missing data in steps of 5%.• Repeat each experiment several times. 25© Fraunhofer IOSB
  26. 26. Evaluation Original, F-Score unmodified Gap size 5 seconds. Precision RecallGap size Gap size5 seconds. 5 seconds. 26© Fraunhofer IOSB
  27. 27. Evaluation Original, F-Score unmodified Gap size 5 seconds. Precision RecallGap size Gap size5 seconds. 5 seconds. 27© Fraunhofer IOSB
  28. 28. Evaluation ROC-curves for video (d) with gap size 5 seconds. TPR TPR FPR FPR With interpolation and hallucination. Without interpolation and hallucination. 28© Fraunhofer IOSB
  29. 29. Evaluation Video (d), F-Score Gap size 5 seconds.Gap size Gap size5 seconds. 5 seconds. 29© Fraunhofer IOSB
  30. 30. Conclusion• low level: interpolation of data and its uncertainty.  ordinary incomplete data.• high level: extension of the situation recognition inference algorithm (hallucination, abduction).  high-level incomplete data such as occlusions.• Knowledge base for vehicle-centered situations.• Runs in real-time on off-the-shelf hardware. 30© Fraunhofer IOSB
  31. 31. Finis. 31© Fraunhofer IOSB

×