Your SlideShare is downloading. ×
Ontology Based Object Learning and Recognition
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Ontology Based Object Learning and Recognition

1,792
views

Published on

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,792
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
51
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Ontology Based Object Learning and Recognition PhD Defence 14/12/2005 Supervised by Monique Thonnat Nicolas MAILLOT Orion team INRIA Sophia Antipolis
  • 2.
    • Introduction
    • State of the Art
    • Knowledge Acquisition
    • Visual Concept Learning
    • Object Categorization
    • Results
    • Conclusion
    Outline
  • 3. Introduction
    • Context : Semantic image interpretation
    • Goal : Object recognition
    • More precisely : object categorization (i.e. finding the category of an object) and not object identification (i.e. recognition of an individual)
    • Approach : Cognitive vision techniques [ECVision Roadmap 04]
    • Mixing knowledge representation, machine learning, image processing and reasoning techniques
  • 4. Introduction: Semantic Image Interpretation Oslo Accords (1993)
    • Semantics is not inside the image:
    handshake agreement Need of a priori knowledge in international politics
  • 5. Introduction: object categorization
    • Assigning a category (e.g. Aircraft, Galaxy) to a region of the image
    • Categories are discrete entities characterized by properties shared by their members
    Aircraft
  • 6. Introduction: Goal
    • Issues:
      • Knowledge acquisition
      • Semantic gap
      • Use of acquired knowledge for performing object categorization
    • Goal: Enabling experts to build object categorization systems dedicated to his/her domain of interest (e.g. biology)
    • Restricted scope:
      • One main object per image
      • Need of a well-defined expertise
  • 7. Introduction: Proposed Approach
    • Decomposition of the object categorization problem in three levels of abstraction:
    High-Level Interpretation Mapping Image Processing Domain knowledge Knowledge about the mapping between domain knowledge and image processing knowledge
  • 8. Introduction: Proposed Approach
    • Use of ontological engineering combined with machine learning techniques
    Reduction of the knowledge acquisition problem and of the semantic gap Performing categorization as experts do
  • 9.
    • Introduction
    • State of the Art
    • Knowledge Acquisition
    • Visual Concept Learning
    • Object Categorization
    • Results
    • Conclusion
    Outline
  • 10. State of the Art : Object Recognition
    • [Brooks83] Object modeling by ribbons. Geometric reasoning
    • [Havaldar96] use of qualitative geometric relationships (e.g. proximity, symmetry)
    • [Basri96] Combination of alignment method with recognition by prototypes
    • [Sangineto03] Recognition based on the shape invariants of object categories
    Geometric model alignment
    • Geometric Methods
  • 11. State of the Art : Object Recognition
    • Appearance-Based Methods
    Implicit objects models Use of multiple views
        • [Swain and Ballard 91] Objects represented by color histograms
        • [Schmid97] Local features. Introduction of a voting algorithm
        • [Schiele00] Receptive field histograms for approximating the local appearance
        • [Fergus03] Local features. Objects modeled as constellations of parts
  • 12. State of the Art : Object Recognition
    • Knowledge-Based Methods [Crevier97]
    • [Draper89] Blackboard architecture. Schemas ( frames + procedures ). Hypothesis generation/verification.
    • [Matsuyama90] T hree expert systems . Frames + rules . Both model driven and data driven. hypotheses generation/verification.
    • [Hudelot05] Cooperation between three knowledge-based systems ( Frames + rules) . Data management functionalities.
  • 13. State of the Art : Object Recognition
    • Summary:
      • Geometric Methods
        • + Strong theoretical foundations
        • - identification of individuals and not categorization
        • - Reliable Extraction of geometric primitives is very difficult
      • Appearance-Based Methods
        • + Effective
        • - Need of large number of samples
        • - Lack of explicitness
      • Knowledge-Based Methods
        • + Explicit
        • + Separation between knowledge and reasoning
      • - Knowledge acquisition bottleneck (mapping knowledge is difficult to acquire)
  • 14.
    • Introduction
    • State of the Art
    • Knowledge Acquisition
    • Visual Concept Learning
    • Object Categorization
    • Results
    • Conclusion
  • 15. Knowledge Acquisition Domain Expert Knowledge Acquisition Knowledge Base Knowledge acquisition guided by a visual concept ontology (i.e geometry, texture, color ) to describe the objects of the domain. Visual Concept Ontology
  • 16. Knowledge Acquisition
    • Ontology
      • Definition: An explicit specification of a conceptualization [Gruber93]
      • Composed of:
        • A set of concepts
        • A set of relations between concepts
        • A set of axioms (e.g. transitivity, reflexivity)
      • Ontological Commitment [Bachimont2000]
        • Shared reference to align with
  • 17. Knowledge Acquisition
    • Visual Concept Ontology
      • 144 concepts :
        • spatial concepts (geometry, size, position, orientation)
        • color concepts (hue, brightness, saturation)
        • texture concepts (pattern, contrast, repartition)
      • Object classes are described by visual concepts
  • 18. Knowledge Acquisition Texture Repartition Pattern Repetitive Random Regular Oriented Granulated Coarse Complex Visual concept ontology content: some texture concepts Based on cognitive experiments [Bhushan et al 97]
  • 19. Knowledge Acquisition Subpart Tree
    • Poaceae :
    • Circular Shape
    • Granulated Texture
    • Pink Color
    Cytoplasm
    • Pore:
    • Subpart of Poaceae
    • Elliptic Shape
    • Small Size
    Domain knowledge described using visual concept ontology Poaceae Pollen Pore
  • 20.
    • Knowledge Formalization
      • Domain class hierarchy: from general to specialized classes
      • Domain Partonomy: subparts linked to domain classes
      • Class: a category (e.g. aircraft, pollen grain ) described by visual concepts
      • Representation by frames with slots
    Knowledge Acquisition
  • 21. Knowledge Acquisition Each visual concept is associated with numerical features: Histograms Color Coherence Vectors [Pass96] Blue, Bright, Dark Color Gabor Features [Manjunath 96] Co-Occurrence Matrices Granulated, Smooth Texture SIFT Features [Lowe 99] Polygonal, Straight Shape Numerical Features Examples Visual Concept
  • 22. Knowledge Acquisition
    • Importance of acquisition context
      • Visual description is valid for an image acquisition context
    Acquisition Context Point of View Sensor Rear View Front View Profile View Microscope Camera CCD Camera IR Camera
  • 23. Domain class hierarchy Subparts hierarchy Ontology driven description Image samples management Knowledge Acquisition
  • 24. Poaceae Composition Link Specialization Link Pollen Grain Pori Non Apertured Pollen Cupressaceae Pori of Poaceae Pori of Parietaria Knowledge Base (18 domain classes + 17 visual concepts) Cytoplasm Of Cupressaceae Pollen with Pori Pollen with Pori and Colpi Apertured Pollen Parietaria Olea Colpi Colpi of Olea Knowledge Acquisition Context: Sensor: Microscope Magnification: 60 Dye: Fuchsin
  • 25. High-Level Interpretation Mapping Image Processing Domain knowledge Completely Acquired Mapping Knowledge Partially Acquired Knowledge Acquisition
    • Conclusion:
  • 26.
    • Introduction
    • State of the Art
    • Knowledge Acquisition
    • Visual Concept Learning
    • Object Categorization
    • Results
    • Conclusion
    Talk Overview
  • 27. Visual Concept Learning
    • Visual Concept Learning
      • Goal: Producing visual concept detectors
      • Why: Mapping knowledge is difficult to acquire
      • How: Training of Support Vector Machines (SVM) with annotated samples
    Granulated Texture Detector Granulated Texture Confidence=0.8
  • 28. Visual Concept Learning
    • Image Sample Segmentation and Annotation using visual concepts
    • Three Approaches:
      • Manual approach
      • Use of 3-D models
      • Weakly-supervised approach
  • 29. Selection of an image sample of Poaceae object Interactive selection of region of interest with a drawing tool
    • Image Sample Segmentation and Annotation: Manual Approach
    • Annotation of selected region by visual concepts:
      • - Pink
      • Large
      • Circular
    Visual Concept Learning
  • 30. Visual Concept Learning
    • Image Sample Segmentation and Annotation: Use of 3-D Models (meshes)
  • 31. Automatic Segmentation Feature Extraction Clustering (k-means) Cluster Visualization and Annotation Visual Concept Learning
    • Image Sample Segmentation and Annotation: weakly-supervised approach
    Image training set Annotated Clusters Visual concept Ontology
  • 32. Automatic Segmentation Size Computation k-means Small Cluster Visualization and Annotation
    • Example: clustering for visual concept category Size
    Visual concept Ontology Visual Concept Learning Image Training Set … … … … … Large
  • 33.
    • Learning (for each visual concept C used during knowledge acquisition)
    Get Positive and Negative Samples Of C Visual Concept Detector SVM Training Feature Extraction And Selection Annotated Regions Visual Concept Learning SVM based on Radial Basis Function Kernels
  • 34. Granulated Texture Detector
    • Example: Learn the visual concept Granulated Texture
      • Visual concept detectors are used to complete the mapping knowledge
    Get Positive and Negative Samples of Concept Granulated Texture Annotated Regions Visual Concept Learning LDA SVM Gabor Filter
  • 35.
    • Introduction
    • State of the Art
    • Knowledge Acquisition
    • Visual Concept Learning
    • Object Categorization
    • Results
    • Conclusion
    Talk Overview
  • 36. Object Categorization
    • Object categorization based on:
      • Acquired knowledge (domain knowledge + mapping knowledge)
      • Visual concept detectors
    • Mechanism: Hypothesis Generation/Verification
    Object Categorization Input Image Class + Visual Description
  • 37.
    • Algorithm: Hierarchical exploration of object classes
    • For each class of the class hierarchy (from root class)
      • Hypothesis generation: generation of a set of hypothetic visual concepts
      • Visual detection of the hypothetic visual concepts in the segmented image
      • Recursion on sub-parts
      • Hypothesis verification: object/class matching w.r.t. a matching threshold
      • If the class is verified then consider sub-classes
    Object Categorization
  • 38.
    • Matching (matching threshold=0.5)
    Circular Shape Detector Granulated Texture Detector Pink Hue Detector 0.63 Σ Object Categorization 0.5 0.6 0.8 (0.5+0.6+0.8)/3 0.63>0.5 : hypothesis verified ? Feature Extraction Automatic Segmentation
    • Poaceae :
    • Circular Shape
    • Granulated Texture
    • Pink Hue
    Current Hypothesis :
  • 39. Object Categorization Automatic Segmentation Feature Extraction Input Image Poaceae 0.63 Circular 0.5 Pink 0.8 Granulated 0.6 Object Categorization Visual Concept Detectors Mapping Knowledge Base
  • 40.
    • Introduction
    • State of the Art
    • Knowledge Acquisition
    • Visual Concept Learning
    • Object Categorization
    • Results
    • Conclusion
    Talk Overview
  • 41. Results
    • Application: Semantic image indexing and retrieval
    • Domain: Transport Vehicles (aircrafts, motorbikes, cars) in their environment
    • Goal: Enabling Retrieval/Indexing by concept
      • User-friendliness
      • Efficiency: no need to store pre-computed feature vectors
    • Issue: trade-off between semantic richness and amount of work needed to build semantic indexing and retrieval systems
  • 42. Results
    • Semantic Indexing
    Image Database Object Categorization Indexed Images Use of categorization results as index for images Indexing time: 1 sec for a 600x400 image on a Intel Pentium IV 3.06Ghz
  • 43. Results
    • Query by concept (opposed to query by example):
    Indexed Images Semantic Query: Object Class / Object Description
    • Example of semantic queries: “ Aircraft ”, “ Gray Aircraft and Blue Sky ”
    Retrieved Images Retrieval
  • 44. Results
    • [Fauqueur03] Retrieval/Indexing based on region templates
    • [Town04] Supervised learning used for mapping image data to a domain ontology
    • [Mezaris04] Querying based on an object ontology (color, position, size, shape). Machine learning and user feedback are used for improving system efficiency
    No approach combines weak supervision with a rich high-level knowledge layer
  • 45. Results Composition Link Specialization Link Outdoor Scene Transport Vehicles Background Sky Aircraft Tarmac Grass Sea Car Motorbike Knowledge acquisition
  • 46. Results Knowledge acquisition Uniform Bottom Green Grass Uniform Bottom Grey Black Tarmac Smooth Top Dark Light Blue Grey Sky Center Polygonal Motorbike Center Polygonal Car Center Polygonal Aircraft Pattern Position Geometry Brightness Hue
  • 47. Results
    • Use of the Caltech image database
    • Training Set : 850 images (aircraft, car, motorbike)
    • Test Set : 2000 images (contains 300 images of each class and 800 background images)
    Background images Images containing objects of interest
  • 48. Results: Caltech Database on 3 object classes
    • Precision/Recall curve
    Precision=n/A Recall=n/N n: number of relevant retrieved images A: number of retrieved images N: number of relevant images
  • 49.
    • Introduction
    • State of the Art
    • Knowledge Acquisition
    • Visual Concept Learning
    • Object Categorization
    • Results
    • Conclusion
    Talk Overview
  • 50. Conclusion
    • Approach: Use of ontological engineering combined with machine learning techniques
    • Three phases:
      • Knowledge acquisition
      • Visual concept learning
      • Object Categorization
    • Applications:
      • Semantic image indexing and retrieval
      • Knowledge acquisition in the domain of palynology
  • 51.
    • Contributions:
      • An extensible and reusable visual concept ontology [maillot04]
        • 144 visual concepts (color, texture and spatial concepts)
      • Original combination of knowledge and learning techniques for explicit domain knowledge elicitation and automatic visual concept detector learning [maillot04]
        • In particular, no inference rules to define for mapping
      • A weakly-supervised annotation approach [maillot05]
        • enables easy image sample annotation
      • An object categorization algorithm [maillot05]
        • reproduces the way expert reason
        • independent of the application domain
    Conclusion
  • 52. Conclusion
    • Strengths and Weaknesses:
      • + Elicitation of domain knowledge
      • + Reduction of the knowledge acquisition bottleneck
      • + Reduction of the semantic gap
      • - Spatial reasoning missing
      • - Image processing algorithms not adaptive
      • - Geometric models not used during categorization
  • 53. Future Works
    • Short-term:
      • Integration in a cognitive vision platform [Hudelot 05]
        • data management
        • top-down and bottom-up mechanisms
        • spatial reasoning
      • Learning for adaptive image segmentation [Martin et al. 06]
    • Long-term:
      • Extension to video content (e.g. temporal concepts)
      • Dynamic knowledge bases (no closed-world assumption)
      • Use of 3-D models for categorization
  • 54.
    • Thank you for your attention
  • 55. Publications
    • [1] Ontology Based Complex Object Recognition
    • N. Maillot,  M. Thonnat Image and Vision Computing Journal Under Minor Revision [2] Towards Ontology Based Cognitive Vision (Long Version)
    • N. Maillot,  M. Thonnat, A. Boucher Machine Vision and Applications Journal (MVA)
    • Springer-Verlag Heidelberg, December 2004, 16(1), pp 33--40 [3] A Weakly Supervised Approach for Semantic Image Indexing and Retrieval  
    • N. Maillot,  M. Thonnat International Conference on Image and Video Retrieval (CIVR  2005)
    • Singapore, 20-22 July 2005 [4] Ontology Based Object Learning and Recognition : Application to Image Retrieval 
    • N. Maillot,  M. Thonnat, C.Hudelot 16th IEEE International Conference on Tools for Artificial Intelligence (ICTAI 2004)
    • Boca Raton, Florida, 15-17 November 2004
    • [5] Towards Ontology Based Cognitive Vision
    • N. Maillot,  M. Thonnat, A. Boucher Third International Conference on Computer Vision Systems (ICVS 2003)
    • Graz, Austria, April 2003, LNCS 2626, pp.44-53, Springer-Verlag Berlin Heidelberg 2003
  • 56. Proposed Approach Data Management Knowledge Base of Visual Concepts and Data Data Management Engine Interpretation Knowledge Base of Application Domain and Visual Concepts Interpretation Engine Program Supervision Library of vision programs Knowledge Base of Program Utilization Program Supervision Engine Current Image Interpretation Object Hypotheses Image Processing Request Numerical data Image description Visual Concept Ontology Cognitive vision platform [Hudelot 05]