Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ontology Based Object Learning and Recognition


Published on

Published in: Technology
  • Be the first to comment

Ontology Based Object Learning and Recognition

  1. 1. Ontology Based Object Learning and Recognition PhD Defence 14/12/2005 Supervised by Monique Thonnat Nicolas MAILLOT Orion team INRIA Sophia Antipolis
  2. 2. <ul><li>Introduction </li></ul><ul><li>State of the Art </li></ul><ul><li>Knowledge Acquisition </li></ul><ul><li>Visual Concept Learning </li></ul><ul><li>Object Categorization </li></ul><ul><li>Results </li></ul><ul><li>Conclusion </li></ul>Outline
  3. 3. Introduction <ul><li>Context : Semantic image interpretation </li></ul><ul><li>Goal : Object recognition </li></ul><ul><li>More precisely : object categorization (i.e. finding the category of an object) and not object identification (i.e. recognition of an individual) </li></ul><ul><li>Approach : Cognitive vision techniques [ECVision Roadmap 04] </li></ul><ul><li>Mixing knowledge representation, machine learning, image processing and reasoning techniques </li></ul>
  4. 4. Introduction: Semantic Image Interpretation Oslo Accords (1993) <ul><li>Semantics is not inside the image: </li></ul>handshake agreement Need of a priori knowledge in international politics
  5. 5. Introduction: object categorization <ul><li>Assigning a category (e.g. Aircraft, Galaxy) to a region of the image </li></ul><ul><li>Categories are discrete entities characterized by properties shared by their members </li></ul>Aircraft
  6. 6. Introduction: Goal <ul><li>Issues: </li></ul><ul><ul><li>Knowledge acquisition </li></ul></ul><ul><ul><li>Semantic gap </li></ul></ul><ul><ul><li>Use of acquired knowledge for performing object categorization </li></ul></ul><ul><li>Goal: Enabling experts to build object categorization systems dedicated to his/her domain of interest (e.g. biology) </li></ul><ul><li>Restricted scope: </li></ul><ul><ul><li>One main object per image </li></ul></ul><ul><ul><li>Need of a well-defined expertise </li></ul></ul>
  7. 7. Introduction: Proposed Approach <ul><li>Decomposition of the object categorization problem in three levels of abstraction: </li></ul>High-Level Interpretation Mapping Image Processing Domain knowledge Knowledge about the mapping between domain knowledge and image processing knowledge
  8. 8. Introduction: Proposed Approach <ul><li>Use of ontological engineering combined with machine learning techniques </li></ul>Reduction of the knowledge acquisition problem and of the semantic gap Performing categorization as experts do
  9. 9. <ul><li>Introduction </li></ul><ul><li>State of the Art </li></ul><ul><li>Knowledge Acquisition </li></ul><ul><li>Visual Concept Learning </li></ul><ul><li>Object Categorization </li></ul><ul><li>Results </li></ul><ul><li>Conclusion </li></ul>Outline
  10. 10. State of the Art : Object Recognition <ul><li>[Brooks83] Object modeling by ribbons. Geometric reasoning </li></ul><ul><li>[Havaldar96] use of qualitative geometric relationships (e.g. proximity, symmetry) </li></ul><ul><li>[Basri96] Combination of alignment method with recognition by prototypes </li></ul><ul><li>[Sangineto03] Recognition based on the shape invariants of object categories </li></ul>Geometric model alignment <ul><li>Geometric Methods </li></ul>
  11. 11. State of the Art : Object Recognition <ul><li>Appearance-Based Methods </li></ul>Implicit objects models Use of multiple views <ul><ul><ul><li>[Swain and Ballard 91] Objects represented by color histograms </li></ul></ul></ul><ul><ul><ul><li>[Schmid97] Local features. Introduction of a voting algorithm </li></ul></ul></ul><ul><ul><ul><li>[Schiele00] Receptive field histograms for approximating the local appearance </li></ul></ul></ul><ul><ul><ul><li>[Fergus03] Local features. Objects modeled as constellations of parts </li></ul></ul></ul>
  12. 12. State of the Art : Object Recognition <ul><li>Knowledge-Based Methods [Crevier97] </li></ul><ul><li>[Draper89] Blackboard architecture. Schemas ( frames + procedures ). Hypothesis generation/verification. </li></ul><ul><li>[Matsuyama90] T hree expert systems . Frames + rules . Both model driven and data driven. hypotheses generation/verification. </li></ul><ul><li>[Hudelot05] Cooperation between three knowledge-based systems ( Frames + rules) . Data management functionalities. </li></ul>
  13. 13. State of the Art : Object Recognition <ul><li>Summary: </li></ul><ul><ul><li>Geometric Methods </li></ul></ul><ul><ul><ul><li>+ Strong theoretical foundations </li></ul></ul></ul><ul><ul><ul><li>- identification of individuals and not categorization </li></ul></ul></ul><ul><ul><ul><li>- Reliable Extraction of geometric primitives is very difficult </li></ul></ul></ul><ul><ul><li>Appearance-Based Methods </li></ul></ul><ul><ul><ul><li>+ Effective </li></ul></ul></ul><ul><ul><ul><li>- Need of large number of samples </li></ul></ul></ul><ul><ul><ul><li>- Lack of explicitness </li></ul></ul></ul><ul><ul><li>Knowledge-Based Methods </li></ul></ul><ul><ul><ul><li>+ Explicit </li></ul></ul></ul><ul><ul><ul><li>+ Separation between knowledge and reasoning </li></ul></ul></ul><ul><ul><li> - Knowledge acquisition bottleneck (mapping knowledge is difficult to acquire) </li></ul></ul>
  14. 14. <ul><li>Introduction </li></ul><ul><li>State of the Art </li></ul><ul><li>Knowledge Acquisition </li></ul><ul><li>Visual Concept Learning </li></ul><ul><li>Object Categorization </li></ul><ul><li>Results </li></ul><ul><li>Conclusion </li></ul>
  15. 15. Knowledge Acquisition Domain Expert Knowledge Acquisition Knowledge Base Knowledge acquisition guided by a visual concept ontology (i.e geometry, texture, color ) to describe the objects of the domain. Visual Concept Ontology
  16. 16. Knowledge Acquisition <ul><li>Ontology </li></ul><ul><ul><li>Definition: An explicit specification of a conceptualization [Gruber93] </li></ul></ul><ul><ul><li>Composed of: </li></ul></ul><ul><ul><ul><li>A set of concepts </li></ul></ul></ul><ul><ul><ul><li>A set of relations between concepts </li></ul></ul></ul><ul><ul><ul><li>A set of axioms (e.g. transitivity, reflexivity) </li></ul></ul></ul><ul><ul><li>Ontological Commitment [Bachimont2000] </li></ul></ul><ul><ul><ul><li> Shared reference to align with </li></ul></ul></ul>
  17. 17. Knowledge Acquisition <ul><li>Visual Concept Ontology </li></ul><ul><ul><li>144 concepts : </li></ul></ul><ul><ul><ul><li>spatial concepts (geometry, size, position, orientation) </li></ul></ul></ul><ul><ul><ul><li>color concepts (hue, brightness, saturation) </li></ul></ul></ul><ul><ul><ul><li>texture concepts (pattern, contrast, repartition) </li></ul></ul></ul><ul><ul><li>Object classes are described by visual concepts </li></ul></ul>
  18. 18. Knowledge Acquisition Texture Repartition Pattern Repetitive Random Regular Oriented Granulated Coarse Complex Visual concept ontology content: some texture concepts Based on cognitive experiments [Bhushan et al 97]
  19. 19. Knowledge Acquisition Subpart Tree <ul><li>Poaceae : </li></ul><ul><li>Circular Shape </li></ul><ul><li>Granulated Texture </li></ul><ul><li>Pink Color </li></ul>Cytoplasm <ul><li>Pore: </li></ul><ul><li>Subpart of Poaceae </li></ul><ul><li>Elliptic Shape </li></ul><ul><li>Small Size </li></ul>Domain knowledge described using visual concept ontology Poaceae Pollen Pore
  20. 20. <ul><li>Knowledge Formalization </li></ul><ul><ul><li>Domain class hierarchy: from general to specialized classes </li></ul></ul><ul><ul><li>Domain Partonomy: subparts linked to domain classes </li></ul></ul><ul><ul><li>Class: a category (e.g. aircraft, pollen grain ) described by visual concepts </li></ul></ul><ul><ul><li>Representation by frames with slots </li></ul></ul>Knowledge Acquisition
  21. 21. Knowledge Acquisition Each visual concept is associated with numerical features: Histograms Color Coherence Vectors [Pass96] Blue, Bright, Dark Color Gabor Features [Manjunath 96] Co-Occurrence Matrices Granulated, Smooth Texture SIFT Features [Lowe 99] Polygonal, Straight Shape Numerical Features Examples Visual Concept
  22. 22. Knowledge Acquisition <ul><li>Importance of acquisition context </li></ul><ul><ul><li>Visual description is valid for an image acquisition context </li></ul></ul>Acquisition Context Point of View Sensor Rear View Front View Profile View Microscope Camera CCD Camera IR Camera
  23. 23. Domain class hierarchy Subparts hierarchy Ontology driven description Image samples management Knowledge Acquisition
  24. 24. Poaceae Composition Link Specialization Link Pollen Grain Pori Non Apertured Pollen Cupressaceae Pori of Poaceae Pori of Parietaria Knowledge Base (18 domain classes + 17 visual concepts) Cytoplasm Of Cupressaceae Pollen with Pori Pollen with Pori and Colpi Apertured Pollen Parietaria Olea Colpi Colpi of Olea Knowledge Acquisition Context: Sensor: Microscope Magnification: 60 Dye: Fuchsin
  25. 25. High-Level Interpretation Mapping Image Processing Domain knowledge Completely Acquired Mapping Knowledge Partially Acquired Knowledge Acquisition <ul><li>Conclusion: </li></ul>
  26. 26. <ul><li>Introduction </li></ul><ul><li>State of the Art </li></ul><ul><li>Knowledge Acquisition </li></ul><ul><li>Visual Concept Learning </li></ul><ul><li>Object Categorization </li></ul><ul><li>Results </li></ul><ul><li>Conclusion </li></ul>Talk Overview
  27. 27. Visual Concept Learning <ul><li>Visual Concept Learning </li></ul><ul><ul><li>Goal: Producing visual concept detectors </li></ul></ul><ul><ul><li>Why: Mapping knowledge is difficult to acquire </li></ul></ul><ul><ul><li>How: Training of Support Vector Machines (SVM) with annotated samples </li></ul></ul>Granulated Texture Detector Granulated Texture Confidence=0.8
  28. 28. Visual Concept Learning <ul><li>Image Sample Segmentation and Annotation using visual concepts </li></ul><ul><li>Three Approaches: </li></ul><ul><ul><li>Manual approach </li></ul></ul><ul><ul><li>Use of 3-D models </li></ul></ul><ul><ul><li>Weakly-supervised approach </li></ul></ul>
  29. 29. Selection of an image sample of Poaceae object Interactive selection of region of interest with a drawing tool <ul><li>Image Sample Segmentation and Annotation: Manual Approach </li></ul><ul><li>Annotation of selected region by visual concepts: </li></ul><ul><ul><li>- Pink </li></ul></ul><ul><ul><li>Large </li></ul></ul><ul><ul><li>Circular </li></ul></ul>Visual Concept Learning
  30. 30. Visual Concept Learning <ul><li>Image Sample Segmentation and Annotation: Use of 3-D Models (meshes) </li></ul>
  31. 31. Automatic Segmentation Feature Extraction Clustering (k-means) Cluster Visualization and Annotation Visual Concept Learning <ul><li>Image Sample Segmentation and Annotation: weakly-supervised approach </li></ul>Image training set Annotated Clusters Visual concept Ontology
  32. 32. Automatic Segmentation Size Computation k-means Small Cluster Visualization and Annotation <ul><li>Example: clustering for visual concept category Size </li></ul>Visual concept Ontology Visual Concept Learning Image Training Set … … … … … Large
  33. 33. <ul><li>Learning (for each visual concept C used during knowledge acquisition) </li></ul>Get Positive and Negative Samples Of C Visual Concept Detector SVM Training Feature Extraction And Selection Annotated Regions Visual Concept Learning SVM based on Radial Basis Function Kernels
  34. 34. Granulated Texture Detector <ul><li>Example: Learn the visual concept Granulated Texture </li></ul><ul><ul><li>Visual concept detectors are used to complete the mapping knowledge </li></ul></ul>Get Positive and Negative Samples of Concept Granulated Texture Annotated Regions Visual Concept Learning LDA SVM Gabor Filter
  35. 35. <ul><li>Introduction </li></ul><ul><li>State of the Art </li></ul><ul><li>Knowledge Acquisition </li></ul><ul><li>Visual Concept Learning </li></ul><ul><li>Object Categorization </li></ul><ul><li>Results </li></ul><ul><li>Conclusion </li></ul>Talk Overview
  36. 36. Object Categorization <ul><li>Object categorization based on: </li></ul><ul><ul><li>Acquired knowledge (domain knowledge + mapping knowledge) </li></ul></ul><ul><ul><li>Visual concept detectors </li></ul></ul><ul><li>Mechanism: Hypothesis Generation/Verification </li></ul>Object Categorization Input Image Class + Visual Description
  37. 37. <ul><li>Algorithm: Hierarchical exploration of object classes </li></ul><ul><li>For each class of the class hierarchy (from root class) </li></ul><ul><ul><li>Hypothesis generation: generation of a set of hypothetic visual concepts </li></ul></ul><ul><ul><li>Visual detection of the hypothetic visual concepts in the segmented image </li></ul></ul><ul><ul><li>Recursion on sub-parts </li></ul></ul><ul><ul><li>Hypothesis verification: object/class matching w.r.t. a matching threshold </li></ul></ul><ul><ul><li>If the class is verified then consider sub-classes </li></ul></ul>Object Categorization
  38. 38. <ul><li>Matching (matching threshold=0.5) </li></ul>Circular Shape Detector Granulated Texture Detector Pink Hue Detector 0.63 Σ Object Categorization 0.5 0.6 0.8 (0.5+0.6+0.8)/3 0.63>0.5 : hypothesis verified ? Feature Extraction Automatic Segmentation <ul><li>Poaceae : </li></ul><ul><li>Circular Shape </li></ul><ul><li>Granulated Texture </li></ul><ul><li>Pink Hue </li></ul>Current Hypothesis :
  39. 39. Object Categorization Automatic Segmentation Feature Extraction Input Image Poaceae 0.63 Circular 0.5 Pink 0.8 Granulated 0.6 Object Categorization Visual Concept Detectors Mapping Knowledge Base
  40. 40. <ul><li>Introduction </li></ul><ul><li>State of the Art </li></ul><ul><li>Knowledge Acquisition </li></ul><ul><li>Visual Concept Learning </li></ul><ul><li>Object Categorization </li></ul><ul><li>Results </li></ul><ul><li>Conclusion </li></ul>Talk Overview
  41. 41. Results <ul><li>Application: Semantic image indexing and retrieval </li></ul><ul><li>Domain: Transport Vehicles (aircrafts, motorbikes, cars) in their environment </li></ul><ul><li>Goal: Enabling Retrieval/Indexing by concept </li></ul><ul><ul><li>User-friendliness </li></ul></ul><ul><ul><li>Efficiency: no need to store pre-computed feature vectors </li></ul></ul><ul><li>Issue: trade-off between semantic richness and amount of work needed to build semantic indexing and retrieval systems </li></ul>
  42. 42. Results <ul><li>Semantic Indexing </li></ul>Image Database Object Categorization Indexed Images Use of categorization results as index for images Indexing time: 1 sec for a 600x400 image on a Intel Pentium IV 3.06Ghz
  43. 43. Results <ul><li>Query by concept (opposed to query by example): </li></ul>Indexed Images Semantic Query: Object Class / Object Description <ul><li>Example of semantic queries: “ Aircraft ”, “ Gray Aircraft and Blue Sky ” </li></ul>Retrieved Images Retrieval
  44. 44. Results <ul><li>[Fauqueur03] Retrieval/Indexing based on region templates </li></ul><ul><li>[Town04] Supervised learning used for mapping image data to a domain ontology </li></ul><ul><li>[Mezaris04] Querying based on an object ontology (color, position, size, shape). Machine learning and user feedback are used for improving system efficiency </li></ul>No approach combines weak supervision with a rich high-level knowledge layer
  45. 45. Results Composition Link Specialization Link Outdoor Scene Transport Vehicles Background Sky Aircraft Tarmac Grass Sea Car Motorbike Knowledge acquisition
  46. 46. Results Knowledge acquisition Uniform Bottom Green Grass Uniform Bottom Grey Black Tarmac Smooth Top Dark Light Blue Grey Sky Center Polygonal Motorbike Center Polygonal Car Center Polygonal Aircraft Pattern Position Geometry Brightness Hue
  47. 47. Results <ul><li>Use of the Caltech image database </li></ul><ul><li>Training Set : 850 images (aircraft, car, motorbike) </li></ul><ul><li>Test Set : 2000 images (contains 300 images of each class and 800 background images) </li></ul>Background images Images containing objects of interest
  48. 48. Results: Caltech Database on 3 object classes <ul><li>Precision/Recall curve </li></ul>Precision=n/A Recall=n/N n: number of relevant retrieved images A: number of retrieved images N: number of relevant images
  49. 49. <ul><li>Introduction </li></ul><ul><li>State of the Art </li></ul><ul><li>Knowledge Acquisition </li></ul><ul><li>Visual Concept Learning </li></ul><ul><li>Object Categorization </li></ul><ul><li>Results </li></ul><ul><li>Conclusion </li></ul>Talk Overview
  50. 50. Conclusion <ul><li>Approach: Use of ontological engineering combined with machine learning techniques </li></ul><ul><li>Three phases: </li></ul><ul><ul><li>Knowledge acquisition </li></ul></ul><ul><ul><li>Visual concept learning </li></ul></ul><ul><ul><li>Object Categorization </li></ul></ul><ul><li>Applications: </li></ul><ul><ul><li>Semantic image indexing and retrieval </li></ul></ul><ul><ul><li>Knowledge acquisition in the domain of palynology </li></ul></ul>
  51. 51. <ul><li>Contributions: </li></ul><ul><ul><li>An extensible and reusable visual concept ontology [maillot04] </li></ul></ul><ul><ul><ul><li>144 visual concepts (color, texture and spatial concepts) </li></ul></ul></ul><ul><ul><li>Original combination of knowledge and learning techniques for explicit domain knowledge elicitation and automatic visual concept detector learning [maillot04] </li></ul></ul><ul><ul><ul><li>In particular, no inference rules to define for mapping </li></ul></ul></ul><ul><ul><li>A weakly-supervised annotation approach [maillot05] </li></ul></ul><ul><ul><ul><li>enables easy image sample annotation </li></ul></ul></ul><ul><ul><li>An object categorization algorithm [maillot05] </li></ul></ul><ul><ul><ul><li>reproduces the way expert reason </li></ul></ul></ul><ul><ul><ul><li>independent of the application domain </li></ul></ul></ul>Conclusion
  52. 52. Conclusion <ul><li>Strengths and Weaknesses: </li></ul><ul><ul><li>+ Elicitation of domain knowledge </li></ul></ul><ul><ul><li>+ Reduction of the knowledge acquisition bottleneck </li></ul></ul><ul><ul><li>+ Reduction of the semantic gap </li></ul></ul><ul><ul><li>- Spatial reasoning missing </li></ul></ul><ul><ul><li>- Image processing algorithms not adaptive </li></ul></ul><ul><ul><li>- Geometric models not used during categorization </li></ul></ul>
  53. 53. Future Works <ul><li>Short-term: </li></ul><ul><ul><li>Integration in a cognitive vision platform [Hudelot 05] </li></ul></ul><ul><ul><ul><li>data management </li></ul></ul></ul><ul><ul><ul><li>top-down and bottom-up mechanisms </li></ul></ul></ul><ul><ul><ul><li>spatial reasoning </li></ul></ul></ul><ul><ul><li>Learning for adaptive image segmentation [Martin et al. 06] </li></ul></ul><ul><li>Long-term: </li></ul><ul><ul><li>Extension to video content (e.g. temporal concepts) </li></ul></ul><ul><ul><li>Dynamic knowledge bases (no closed-world assumption) </li></ul></ul><ul><ul><li>Use of 3-D models for categorization </li></ul></ul>
  54. 54. <ul><li>Thank you for your attention </li></ul>
  55. 55. Publications <ul><li>[1] Ontology Based Complex Object Recognition </li></ul><ul><li>N. Maillot,  M. Thonnat Image and Vision Computing Journal Under Minor Revision [2] Towards Ontology Based Cognitive Vision (Long Version) </li></ul><ul><li>N. Maillot,  M. Thonnat, A. Boucher Machine Vision and Applications Journal (MVA) </li></ul><ul><li>Springer-Verlag Heidelberg, December 2004, 16(1), pp 33--40 [3] A Weakly Supervised Approach for Semantic Image Indexing and Retrieval   </li></ul><ul><li>N. Maillot,  M. Thonnat International Conference on Image and Video Retrieval (CIVR  2005) </li></ul><ul><li>Singapore, 20-22 July 2005 [4] Ontology Based Object Learning and Recognition : Application to Image Retrieval  </li></ul><ul><li>N. Maillot,  M. Thonnat, C.Hudelot 16th IEEE International Conference on Tools for Artificial Intelligence (ICTAI 2004) </li></ul><ul><li>Boca Raton, Florida, 15-17 November 2004 </li></ul><ul><li>[5] Towards Ontology Based Cognitive Vision </li></ul><ul><li>N. Maillot,  M. Thonnat, A. Boucher Third International Conference on Computer Vision Systems (ICVS 2003) </li></ul><ul><li>Graz, Austria, April 2003, LNCS 2626, pp.44-53, Springer-Verlag Berlin Heidelberg 2003 </li></ul>
  56. 56. Proposed Approach Data Management Knowledge Base of Visual Concepts and Data Data Management Engine Interpretation Knowledge Base of Application Domain and Visual Concepts Interpretation Engine Program Supervision Library of vision programs Knowledge Base of Program Utilization Program Supervision Engine Current Image Interpretation Object Hypotheses Image Processing Request Numerical data Image description Visual Concept Ontology Cognitive vision platform [Hudelot 05]