EnContRAEngine for Content-Based Retrieval Approaches                                 Ricardo José São Pedro Dias         ...
ContextColaDI Project  – Platform for Project Collaboration in Industrial    DesignPeriod: March 2010 / November 2011(EnCo...
ObjectivesGeneral Framework for (Content-Based)Retrieval Approaches and Applications  – Features:     •   Indexing     •  ...
Advantages for Development1. Modularity2. Easy to use – Low learning curve3. Fast development of new approaches  – Example...
Multimedia SupportSupport for different multimedia types  – Pictures  – Drawings  – 3D Objects  – Audio / Music
Typical Application Architecture
EnContRA Modules
Indexing and Retrieving Pictures using QBECREATING A SIMPLE APPROACH
ObjectivesCreate a simple Query by Example ImageRetrieval Application
Data Model – Input Data
Objective     Query                    QBEExtracts: Scalable Color
Pieces to Assembly1. Image Descriptor: Scalable Color2. Indexing Structure: In Memory Simple Index3. Searching Algorithm: ...
Pieces to Assembly1. Image Descriptor: Scalable Color2. Indexing Structure: In Memory Simple Index3. Searching Algorithm: ...
Choosing a Feature to be extractScalable Color Descriptor (Extractor)DescriptorExtractor extractor = new                  ...
Pieces to Assembly1. Image Descriptor: Scalable Color2. Indexing Structure: In Memory Simple Index3. Searching Algorithm: ...
Choosing an Indexing StructureIn Memory Simple IndexAbstractIndex index = new SimpleIndex(); Descriptor               Desc...
Pieces to Assembly1. Image Descriptor: Scalable Color2. Indexing Structure: In Memory Simple Index3. Searching Algorithm: ...
SearchingLinear Search (for now!)Searcher searcher = new SimpleSearcher<IndexedObject>(); Descriptor              Descriptor
Assembling all the components                 Searcher searcher = new SimpleSearcher<IndexedObject>(); Setting Main       ...
Indexing the DatasetFile [] pictures = getFilePictures(dataset);for (File pic : pictures) {     BufferedImage image = Imag...
Performing a Search – Similar//Load the image queryBufferedImage image = readQuery();//Perform the search using similarRes...
Performing a Search – Query API            CriteriaBuilderImpl cb = new CriteriaBuilderImpl();            Path<IndexedObje...
Indexing and Retrieving Pictures using QBEA MORE COMPLEX APPROACH
ObjectivesCreate a Drawing Retrieval Application, byemploying Query By Example (or Sketch)  – Queries:     • 2D Drawings (...
Input ModelDrawing Modelpublic class DrawingModel implements IEntity<Long> {          …          private Drawing drawing; ...
Model to IndexedObject                                                 CEDD IdxObj                                        ...
Pieces to Assembly1. Descriptor: Image Descriptors + TopoGeo2. Indexing Structure: NBTree3. Searching Algorithm: NBTree Se...
Pieces to Assembly1. Descriptor: Image Descriptors + TopoGeo2. Indexing Structure: NBTree3. Searching Algorithm: NBTree Se...
DescriptorsImage DescriptorsDescriptorExtractor ceddExtractor = new CEDDDescriptor<IndexedObject>();DescriptorExtractor ed...
Pieces to Assembly1. Descriptor: Image Descriptors + TopoGeo2. Indexing Structure: NBTree3. Searching Algorithm: NBTree Se...
BTree for IndexingParameters:    – the name of the index    – the type of objects to be indexed (class)BTreeIndex exampleI...
Pieces to Assembly1. Descriptor: Image Descriptors + TopoGeo2. Indexing Structure: NBTree3. Searching Algorithm: NBTree Se...
NBTree SearcherTwo flavors:  – Regular (original)  AbstractSearcher searcher = new NBTreeSearcher();  – Parallel  To spee...
Picture – Composed SearchingAbstractSearcher imageSearcher = new ImageSearchEngine();imageSearcher.setQueryProcessor(new Q...
Drawing SearcherAbstractSearcher vectSearcher = new ParallelNBTreeSearcher();vectSearcher.setQueryProcessor( new QueryProc...
Performing a Search – Individual            CriteriaQuery<DrawingModel> query =                                       cb.c...
Performing a Search – Combined            Similar simD = cb.similar(drawingPath, new IndexedObject(drawing));            S...
Custom IndexedObject Factorypublic class ImageIndexedObjectFactory extends AnnotatedIndexedObjectFactory{          …      ...
Custom Image Searcherpublic class ImageSearchEngine implements AbstractSearcher<Long> {          …          protected List...
Some features of the Query APIQUERY API
Operators•   AND/ OR•   EQUAL / NOT EQUAL•   SIMILAR•   NOT
Query Processors• Cascade Processor  – Each sub-expression at a time• Parallel Processor  – Optimization for sub-expressio...
Some demos developed during the projectDEMOS
DemosAvailable at http://www.youtube.com/inevopt   Image & Vectorial Search           Android Visual Search
How to get EnContRA, and more documentation and support?GETTING ENCONTRA
Checkout/Push Source CodeCheckout  git clone git@inevo.sourcerepo.com:inevo/encontra.gitCommit and Push  git commit –m “+ ...
Contributing and Compiling Modulesmvn install full deploy (compile, package, run tests)mvn package full deploy (compile,...
Documentation & Support• EnContRA 101 – Dev Tutorial (almost finished)• Javadocs• Source code• People:  – Me ricardo.dias...
EnContRAEngine for Content-Based Retrieval Approaches                           The End                                 Ri...
Encontra presentation
Upcoming SlideShare
Loading in …5
×

Encontra presentation

518 views
461 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
518
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Encontra presentation

  1. 1. EnContRAEngine for Content-Based Retrieval Approaches Ricardo José São Pedro Dias 07/02/2012
  2. 2. ContextColaDI Project – Platform for Project Collaboration in Industrial DesignPeriod: March 2010 / November 2011(EnContRA – until February 2011)
  3. 3. ObjectivesGeneral Framework for (Content-Based)Retrieval Approaches and Applications – Features: • Indexing • Features Extraction • Searching / Retrieval Algorithms • Extensible Query Processing
  4. 4. Advantages for Development1. Modularity2. Easy to use – Low learning curve3. Fast development of new approaches – Examples: • A new descriptor • A new searching / retrieval algorithm • A new indexing structure • Etc.
  5. 5. Multimedia SupportSupport for different multimedia types – Pictures – Drawings – 3D Objects – Audio / Music
  6. 6. Typical Application Architecture
  7. 7. EnContRA Modules
  8. 8. Indexing and Retrieving Pictures using QBECREATING A SIMPLE APPROACH
  9. 9. ObjectivesCreate a simple Query by Example ImageRetrieval Application
  10. 10. Data Model – Input Data
  11. 11. Objective Query QBEExtracts: Scalable Color
  12. 12. Pieces to Assembly1. Image Descriptor: Scalable Color2. Indexing Structure: In Memory Simple Index3. Searching Algorithm: Simple Searcher
  13. 13. Pieces to Assembly1. Image Descriptor: Scalable Color2. Indexing Structure: In Memory Simple Index3. Searching Algorithm: Simple Searcher
  14. 14. Choosing a Feature to be extractScalable Color Descriptor (Extractor)DescriptorExtractor extractor = new ScalableColorDescriptor<IndexedObject>(); Extractor Descriptor
  15. 15. Pieces to Assembly1. Image Descriptor: Scalable Color2. Indexing Structure: In Memory Simple Index3. Searching Algorithm: Simple Searcher
  16. 16. Choosing an Indexing StructureIn Memory Simple IndexAbstractIndex index = new SimpleIndex(); Descriptor Descriptor
  17. 17. Pieces to Assembly1. Image Descriptor: Scalable Color2. Indexing Structure: In Memory Simple Index3. Searching Algorithm: Simple Searcher
  18. 18. SearchingLinear Search (for now!)Searcher searcher = new SimpleSearcher<IndexedObject>(); Descriptor Descriptor
  19. 19. Assembling all the components Searcher searcher = new SimpleSearcher<IndexedObject>(); Setting Main searcher.setDescriptorExtractor(extractor); Properties searcher.setIndex(index); searcher.setObjectStorage(new Not required SimpleObjectStorage(IndexedObject.class));(recommended) searcher.setResultProvider(new DefaultResultProvider());
  20. 20. Indexing the DatasetFile [] pictures = getFilePictures(dataset);for (File pic : pictures) { BufferedImage image = ImageIO.read(pic); searcher.insert(new IndexedObject(image));}Extracts descriptors and indexes them!
  21. 21. Performing a Search – Similar//Load the image queryBufferedImage image = readQuery();//Perform the search using similarResultSet<IndexedObject> results = searcher.similar(new IndexedObject(image), 20);//Print the ResultsprintResults(results);
  22. 22. Performing a Search – Query API CriteriaBuilderImpl cb = new CriteriaBuilderImpl(); Path<IndexedObject> modelPath = new Query Path<IndexedObject>(IndexedObject.class);Building Similar similar = cb.similar(modelPath, new IndexedObject(image)); CriteriaQuery query = cb.createQuery().where(similar).limit(20);Searching ResultSet<StringObject> results = searcher.search(query);
  23. 23. Indexing and Retrieving Pictures using QBEA MORE COMPLEX APPROACH
  24. 24. ObjectivesCreate a Drawing Retrieval Application, byemploying Query By Example (or Sketch) – Queries: • 2D Drawings (e.g., SVG files) • Pictures
  25. 25. Input ModelDrawing Modelpublic class DrawingModel implements IEntity<Long> { … private Drawing drawing; private BufferedImage image; … @Indexed public BufferedImage getImage(); @Indexed public Drawing getDrawing();}
  26. 26. Model to IndexedObject CEDD IdxObj Image Instance Indexed Edge IdxObj Object ColorL IdxObj Indexed Object Factory DrawingPicture & Vectorial Indexed Drawing Object IdxObj
  27. 27. Pieces to Assembly1. Descriptor: Image Descriptors + TopoGeo2. Indexing Structure: NBTree3. Searching Algorithm: NBTree Searcher
  28. 28. Pieces to Assembly1. Descriptor: Image Descriptors + TopoGeo2. Indexing Structure: NBTree3. Searching Algorithm: NBTree Searcher
  29. 29. DescriptorsImage DescriptorsDescriptorExtractor ceddExtractor = new CEDDDescriptor<IndexedObject>();DescriptorExtractor edgeHistogram = new EdgeHistogramDescriptor<IndexedObject>();DescriptorExtractor colorLayout = new ColorLayoutDescriptor<IndexedObject>();TopoGeo DescriptorTopogeoDescriptorExtractor topogeoDescriptorExtractor = new TopogeoDescriptorExtractor();
  30. 30. Pieces to Assembly1. Descriptor: Image Descriptors + TopoGeo2. Indexing Structure: NBTree3. Searching Algorithm: NBTree Searcher
  31. 31. BTree for IndexingParameters: – the name of the index – the type of objects to be indexed (class)BTreeIndex exampleIndex = new BTreeIndex(“btreeName", Object.class);
  32. 32. Pieces to Assembly1. Descriptor: Image Descriptors + TopoGeo2. Indexing Structure: NBTree3. Searching Algorithm: NBTree Searcher
  33. 33. NBTree SearcherTwo flavors: – Regular (original) AbstractSearcher searcher = new NBTreeSearcher(); – Parallel  To speed the search AbstractSearcher searcher = new ParallelNBTreeSearcher();
  34. 34. Picture – Composed SearchingAbstractSearcher imageSearcher = new ImageSearchEngine();imageSearcher.setQueryProcessor(new QueryProcessorDefaultParallelImpl());imageSearcher.setIndexedObjectFactory(new ImageIndexedObjectFactory());//Creating a combined searcher, with the selected descriptorfor (Map.Entry<String, DescriptorExtractor> entry : availableDescriptors) { AbstractSearcher entrySearcher = new ParallelNBTreeSearcher(); entrySearcher.setQueryProcessor(new QueryProcessorDefaultParallelImpl()); entrySearcher.setIndex(new BTreeIndex("image." + entry.descriptorName, objClass); entrySearcher.setDescriptorExtractor(entry.extractor); imageSearcher.setSearcher("image." + entry.descriptorName, entrySearcher); }searcher.setSearcher("image", imageSearcher);
  35. 35. Drawing SearcherAbstractSearcher vectSearcher = new ParallelNBTreeSearcher();vectSearcher.setQueryProcessor( new QueryProcessorDefaultParallelImpl());vectorialSearcher.setIndex(new BTreeIndex(“vectIndex", TopogeoDescriptor.class));TopogeoDescriptorExtractor topogeoExtractor = new TopogeoDescriptorExtractor();vectorialSearcher.setDescriptorExtractor(topogeoExtractor);searcher.setSearcher("drawing", vectSearcher);
  36. 36. Performing a Search – Individual CriteriaQuery<DrawingModel> query = cb.createQuery(DrawingModel.class); Query Path<DrawingModel> modelPath = query.from(DrawingModel.class);Building Path drawingPath = modelPath.get(“drawing”); Similar similar = cb.similar(drawingPath, new IndexedObject(drawing)); CriteriaQuery query = cb.createQuery().where(similar).limit(20);Searching ResultSet<StringObject> results = searcher.search(query);
  37. 37. Performing a Search – Combined Similar simD = cb.similar(drawingPath, new IndexedObject(drawing)); Similar simP = cb.similar(picturePath, new IndexedObject(image)); Query And andPredicate = cb.and(simD, simP);Building CriteriaQuery query = cb.createQuery() .where(andPredicate ).limit(20);Searching ResultSet<StringObject> results = searcher.search(query);
  38. 38. Custom IndexedObject Factorypublic class ImageIndexedObjectFactory extends AnnotatedIndexedObjectFactory{ … protected List<IndexedObject> createObjects(List<IndexedField> indexedFields) { //create indexedObjects for the DrawingModel instances … } …}
  39. 39. Custom Image Searcherpublic class ImageSearchEngine implements AbstractSearcher<Long> { … protected List<IndexedObject> getIndexedObjects(Object o) throws IndexingException { //create different indexedObjects for the same image, and use //them in different individual searchers } public ResultSet search(Query query) { //create subqueries to perform search in the individual image //searchers } …}
  40. 40. Some features of the Query APIQUERY API
  41. 41. Operators• AND/ OR• EQUAL / NOT EQUAL• SIMILAR• NOT
  42. 42. Query Processors• Cascade Processor – Each sub-expression at a time• Parallel Processor – Optimization for sub-expressions like AND and OR
  43. 43. Some demos developed during the projectDEMOS
  44. 44. DemosAvailable at http://www.youtube.com/inevopt Image & Vectorial Search Android Visual Search
  45. 45. How to get EnContRA, and more documentation and support?GETTING ENCONTRA
  46. 46. Checkout/Push Source CodeCheckout git clone git@inevo.sourcerepo.com:inevo/encontra.gitCommit and Push git commit –m “+ Add: Texture Layout Descriptor.” git push http://schacon.github.com/git/gittutorial.html
  47. 47. Contributing and Compiling Modulesmvn install full deploy (compile, package, run tests)mvn package full deploy (compile, package)mvn –DskipTests=true install full deploy (skip tests to speed up) http://maven.apache.org/
  48. 48. Documentation & Support• EnContRA 101 – Dev Tutorial (almost finished)• Javadocs• Source code• People: – Me ricardo.dias@ist.utl.pt – Tiago Cardoso  tiago.cardoso@inevo.pt – Nelson Silva  nelson.silva@inevo.pt
  49. 49. EnContRAEngine for Content-Based Retrieval Approaches The End Ricardo José São Pedro Dias 07/02/2012

×