Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Part-based Object Retrieval with Binary Partition Trees

Phd thesis co-advised by Ferran Marqués (UPC) and Shih-Fu Chang (Columbia University).

More details in:
https://imatge.upc.edu/web/publications/part-based-object-retrieval-binary-partition-trees

  • Login to see the comments

Part-based Object Retrieval with Binary Partition Trees

  1. 1. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Part-based Object Retrieval with Binary Partition Trees Phd thesis dissertation by Xavier Giró-i-Nieto Supervised by Professor Ferran Marqués Acosta (UPC) and Professor Shih-Fu Chang (Columbia) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 1
  2. 2. Acknowledgements Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Jury DVMM @ Columbia GPI @ UPC Students @ UPC X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 2
  3. 3. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction ← 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 3
  4. 4. Problem statement Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Content-Based Image Retrieval X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 4
  5. 5. Problem statement Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Semantic gap between user query and machine data. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 5
  6. 6. Challenges Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Visual diversity within the same semantic class anchor X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 6
  7. 7. Challenges Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Visual diversity within the same instance This anchor X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 7
  8. 8. Challenges Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Semantics located at multiple scales News Head TV studio Anchor Button Jacket X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 8
  9. 9. Problem statement Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) This thesis focuses in the object retrieval with one example... Query object Target dataset Ranked list X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 9
  10. 10. Problem statement Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) ...and multiple examples. Query objects Target dataset Ranked list X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 10
  11. 11. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions ← Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 11
  12. 12. 2. Hierchical Image Partition Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Object analysis Local scale Sliding windows [Viola-Jones'01] Interest points Image partition SIFT [Lowe'04] SURF [Bay'06] Object Segmentation ? ✘ ✘ X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC ✔ 12
  13. 13. 2. Hierchical Image Partition Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Semantics at multiple scales Multi-scale analysis Spatial Pyramid of points [Lazebnik'06] [Bosch'07] Hierarchical Multiple partition segmentations [Ravinovich'07] [Vieux'10] ✔ [Adamek'06] [Gu'09] X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 13
  14. 14. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 2. Hierchical Image Partition Adopted Solution: Binary Partition Trees (BPTs) [Garrido, Salembier 2000]. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 14
  15. 15. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 2. Hierchical Image Partition Semantic Object “head” Perceptual BPT Split Thesis topic X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 15
  16. 16. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation ← 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 16
  17. 17. 3. Interactive segmentation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Motivations: Ground truth generation and retrieval interface. Graphical Annotation Tool (GAT) Graphical Object Searcher (GOS) Demo @ http://upseek.upc.edu/gat & http://upseek.upc.edu/gos X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 17
  18. 18. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree ← 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 18
  19. 19. 4. Object Tree Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Related work: “Definition Hierarchy”, Jaimes and Chang (2001) Thesis approach BPT nodes BPT leaves X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 19
  20. 20. 4. Object Tree Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Assumption: All objects are connected Interactive segmentation BPT sub-roots selection X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 20
  21. 21. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 4. Object Tree The Object Tree (OT) represents an instance of the object as a hierarchy of regions. User selection of sub-roots on the BPT. Automatic top-down expansion through the BPT X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 21
  22. 22. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features ← Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 22
  23. 23. 5. Region Features Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) MPEG-7 and many more, “Visual Descriptors (VD)” (2001) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 23
  24. 24. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 5. Region Features → Region features in BPT Dominant Colors ➔ Contour Shape ➔ Edge Histogram ➔ NOT Thesis topic X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 24
  25. 25. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features ← Part II 6. Codebooks 7. Fusion of Visual Modalities Part III Evaluation 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 25
  26. 26. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How to evaluate an object retrieval system ? Annotated dataset + Metrics 1. Image Retrieval 2. Object Detection 3. Object Segmentation 4. Search time X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 26
  27. 27. Evaluation (in Part II) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Annotated Dataset ETHZ: Category #1 “Apple logos” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 27
  28. 28. Evaluation (in Part II) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Annotated Dataset ETHZ: Category #2 “Bottles” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 28
  29. 29. Evaluation (in Part II) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Annotated Dataset ETHZ: Category #3 “Giraffes” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 29
  30. 30. Evaluation (in Part II) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Annotated Dataset ETHZ: Category #4 “Mugs” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 30
  31. 31. Evaluation (in Part II) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Annotated Dataset ETHZ: Category #5 “Swans” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 31
  32. 32. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 1. Image Retrieval (global scale) Only global scale is considered. User Query (local) Ranked list of images (global) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 32
  33. 33. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 1. Image Retrieval (global scale) Mean Average Precision (MAP) combines Precision and Recall for a set of queries. Precision Plotted by considering different k's ∣{relevant}∩{retrieved (k )}∣ P (k )= k Recall ∣{relevant }∩{retrieved (k )}∣ R(k )= ∣relevant∣ X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 33
  34. 34. 5. Region features Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Image Retrieval (global scale) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 34
  35. 35. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 2. Object Detection (local scale-rough) User Query (local) Retrieved boxes in relevant images X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 35
  36. 36. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 2. Object Detection (local scale-rough) Pascal criterion requires a minimum of 50% overlapping in the union of detected object box and ground truth box. ∣{correct detections }∣ Detection rate= ∣{relevant }∣ X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 36
  37. 37. 5. Region features Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Object Detection (local scale-rough) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 37
  38. 38. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 3. Object Segmentation (local scale-precise) User Query (local) Retrieved regions from correct detections X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 38
  39. 39. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 3. Object Segmentation (local scale-precise) Jaccard index-J(A,B) compares retrieved region and ground truth mask at the pixel level. ∣A∩B∣ J ( A , B)= ∣A∪B∣ A B Mean Jaccard Index combines the J's obtained by a set of correct detections D. ̄= 1 ∑ J i J ∣D∣ i ∈D X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 39
  40. 40. 5. Region features Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Segmentation (local scale-precise) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 40
  41. 41. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks ← 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 41
  42. 42. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 6. Codebooks Visual Word (VW): Vector quantization of observation (point, region...) onto a codebook [Sivic, Zissermann 2003]. Regions Codebook Code Region Visual Words (a1,a2,a3,a4) (b1,b2,b3,b4) (c1,c2,c3,c4) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 42
  43. 43. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Code Regions (CR) define the basis of the Parts Space. “Regions described by their sub-parts” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 43
  44. 44. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (a) Hard quantization SIFT [Sivic 2003] Regions Codebook Visual Words ✘ (1,0,0,0) ? (1,0,0,0) (0,1,0,0) ? X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC ✘ 44
  45. 45. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (b) Probabilistic quantization [Alpaydin 1998] Regions Codebook Visual Words (0.25,0.25,0.25,0.25) ✘ (1,0,0,0) ✘ (0.25,0.25,0.25,0.25) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 45
  46. 46. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (c) Possibilistic quantization [Bezdek 1995] Regions Codebook Visual Words (0,0,0,0) ✔ (1,0,0,0) ✘ (0,0,0,0) Similarity to CRs X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 46
  47. 47. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How to define the Parts Space with a given codebook ? Possibilistic quantization alone characterizes regions, but not their sub-parts. Region-based Possibilistic X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 47
  48. 48. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How to define the Parts Space with a given codebook ? Bottom-up Max Pooling through BPT ✔ [Boureau'10] Fast Extraction (max) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 48
  49. 49. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) → Hierarchy of Bags of Regions (HBoR) HBoR BPT is exploited to describe sub-trees. Fast Extraction (max) Efficient Analysis (resilency to splits) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 49
  50. 50. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How to generate a codebook ? (1/2) Train set BPT-based decomposition Object parts K-Means Clustering Codebook CR1 CR2 CR3 CR4 X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 50
  51. 51. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How to generate a codebook ? (2/2) a) Foreground and background regions - “generic” b) Only object regions - “frgd” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 51
  52. 52. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 1. Image Retrieval (global scale) - Texture HBoR HBoR features outstand region-based for shape & texture (not dominant colors). Little differences between generic vs frgd codebooks (5 classes in ETHZ). Codebook size X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 52
  53. 53. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features ← Part II 6. Codebooks 7. Fusion of Visual Modalities ← Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 53
  54. 54. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 5. Region Features + 7. Fusion Problem: Similarity scores between descriptors are not comparable. Solution: d color d shape d texture Normalization (z-scores, SVM, w-scores) x ̄color x ̄ shape x ̄texture Fusion (average, product, max/min) x ̄ X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 54
  55. 55. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks ← 7. Fusion of Visual Modalities ← Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 55
  56. 56. 6. Codebooks + 7. Fusion Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) When working with codebooks, when to apply fusion ? (a) Fusion after quantization - “nofused” Query region [ ] color shape texture Target region [ ] color shape texture Separate codebooks Quantization [ ] [ ] VW color VW shape VW texture VW color VW shape VW texture Cosine distance [ ] x ̄color x ̄ shape x ̄ texture Fusion x X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 56
  57. 57. 6. Codebooks + 7. Fusion Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) When working with codebooks, when to apply fusion ? (b) Fusion during quantization - “fused” Query region [ ] color shape texture Target region [ ] color shape texture Fused codebook Quantization [ VW ] [ VW ] Cosine distance x X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 57
  58. 58. 6. Codebooks + 7. Fusion Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) When working with codebooks, when to apply fusion ? 1. Image Retrieval (global scale) z-scores w-scores W-scores outperform z-scores when working with HBoR. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 58
  59. 59. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching ← 9. Part-based Object Model ← Evaluation 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 59
  60. 60. Evaluation (in Part III) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) CCMA News dataset. Topology diversity on 11 anchors objects. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 60
  61. 61. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching ← 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 61
  62. 62. 8. Partition Tree Matching Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Reminder: The Object Retrieval Problem (single instance) Query object Target dataset Ranked list X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 62
  63. 63. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 8. Partition Tree Matching Object Tree (OT) Target BPT Q0 Q1 ? Q2 Q3 Q4 Challenges: (a) Distance for a given OT-BPT correspondence (b) Matching algorithm X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 63
  64. 64. 8. Partition Tree Matching: Distance Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Proposal (ad-hoc): Weighted sum of part matches N d =∑ wi d i , where w i=i i= 0 ∏ Q   j =i⋅w Parent Q  j ∈ Ancestors i i area(i) 1 αi = i i max (1,∣C ∣) area(P ) Children of part i Example: Q0 Q1 Q2 Parents of part i α0=1 /2=w 0 α1=area(Q 1)/area (Q 0) α 2=area(Q 2 )/area(Q0 ) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 64
  65. 65. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Two proposals: (a) Top-down Matching (b) Bottom-Up Matching Object Tree (OT) Target BPT Q0 Q1 ? Q2 Q3 Q4 X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 65
  66. 66. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (a) Top-down Matching Assumption: All OT nodes are contained in a single sub-BPT. -c +c n mo om om mo n Sets a baseline (b) Bottom-Up Matching Assumption: OT nodes are spread through different subBPTs. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 66
  67. 67. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (b) Top-down QT Matching Assumption: The best match for the OT is contained in a single sub-tree of the target BPT. OT Target BPT Q0 1. Set a match for Q0 Q1 Q2 Q0 2. Find matches for Q1 and Q2 Q1 Q2 X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 67
  68. 68. 8. Partition Tree Matching Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) What is the impact of decomposing the query into parts ? Image Retrieval (global scale) MPEG-7 Region Top-down match Retrieval gain until the “correct” amount of parts (3). X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 68
  69. 69. 8. Partition Tree Matching Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) What is the impact of decomposing the query into parts ? Search time MPEG-7 Region Top-down match Exponential growth of search time with the amount of parts. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 69
  70. 70. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (a) Top-down Matching Assumption: All OT nodes are contained in a single sub-BPT. -c +c n mo om om mo n Sets a baseline (b) Bottom-Up Matching Assumption: OT nodes are spread through different subBPTs. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 70
  71. 71. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (c) Bottom-UP QT Matching Q0 1. Set a match for Q1 & Q2 Q1 2. Assess the union of candidates for Q1 and Q2 to match Q0 Q2 This region is not part of the target BPT Q0 Q1 Q2 Weaker assumption that in the top-down case: → OT leaves are among the nodes of the target BPT X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 71
  72. 72. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (c) Bottom-UP QT Matching How to rapidly describe the merged region generated during search time ? HBoR Object Tree (1, 1, 1, 1) Q0 Q1 Q2 → Approximate HBoR satisfies this requirement. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 72
  73. 73. 8. Partition Tree Matching Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) What is the impact of introducing the bottom-up match ? HBoR Bottom-up gain Region VW A clear gain, the object splits in the target BPTs are solved. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 73
  74. 74. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities 8. Partition Tree Matching Part III 9. Part-based Object Model ← 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 74
  75. 75. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Reminder: The Object Retrieval Problem (multiple instances) Query objects Target dataset Ranked list X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 75
  76. 76. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Model-based retrieval Query objects Model Training Target image Retrieved object Test X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 76
  77. 77. 9. Part-based Object Model Training Fusion classifier Inclusion classifier #P ar ts Test Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Parts definition Inclusion classifier #P ar ts Detector classifier Detector classifier Fusion classifier Contextual filtering X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 77
  78. 78. 9. Part-based Object Model Training Fusion classifier Inclusion classifier #P ar ts Test Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Parts definition Inclusion classifier #P ar ts Detector classifier Detector classifier Fusion classifier Contextual filtering X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 78
  79. 79. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Parts definition Unsupervised clustering with Quality Threshold [Heyer 1999]. Two parameters: - Mininum amount of elements in cluster - Maximum radius of a cluster X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 79
  80. 80. 9. Part-based Object Model Training Fusion classifier Inclusion classifier #P ar ts Test Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Parts definition Inclusion classifier #P ar ts Detector classifier Detector classifier Fusion classifier Contextual filtering X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 80
  81. 81. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Inclusion classifier (training) Positive: Parts and their ancestors Negative: Parts' children and positive node's siblings. HBoR X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 81
  82. 82. 9. Part-based Object Model Training Fusion classifier Inclusion classifier #P ar ts Test Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Parts definition Inclusion classifier #P ar ts Detector classifier Detector classifier Fusion classifier Contextual filtering X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 82
  83. 83. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Inclusion classifier (test) Efficient top-down exploration of the target BPT. HBoR Not explored (efficiency) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 83
  84. 84. 9. Part-based Object Model Training Fusion classifier Inclusion classifier #P ar ts Test Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Parts definition Inclusion classifier #P ar ts Detector classifier Detector classifier Fusion classifier Contextual filtering X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 84
  85. 85. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Detector classifier (training) Positive: Parts. Negative: Random sampling (balanced set). Region-based Possibilistic X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 85
  86. 86. 9. Part-based Object Model Training Fusion classifier Inclusion classifier #P ar ts Test Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Parts definition Inclusion classifier #P ar ts Detector classifier Detector classifier Fusion classifier Contextual filtering X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 86
  87. 87. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Fusion classifier (training) Positive training samples represent valid combinations of parts. Negative training samples represent invalid combinations. Parts definition Type 1 Parts of object i (1, 1, 0, 0) Type 2 Type 3 Type 4 Parts of object j (0, 1, 1, 1) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 87
  88. 88. 9. Part-based Object Model Training Fusion classifier Inclusion classifier #P ar ts Test Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Parts definition Inclusion classifier #P ar ts Detector classifier Detector classifier Fusion classifier Contextual filtering X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 88
  89. 89. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How important is a correct estimation of the types of parts ? Image Retrieval Max radius for cluster Min # elems in cluster CodeBook size The performance is dependent from the Quality Threshold parameters. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 89
  90. 90. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) What is the impact of codebook size in terms of time ? Training time Search time Search time does not grow with codebook size (Efficient exploration ??). X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 90
  91. 91. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities 8. Partition Tree Matching Part III 9. Part-based Object Model 10.Conclusions ← X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 91
  92. 92. Conclusions Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) BPT presents several opportunities for local analysis: → Interactive segmentation → Semantic analysis Three main contributions in this thesis: Hierarchy of Bags of Regions: Contribution #1 Contribution #2 ● ● Object Tree - BPT matching ● ● Contribution #3 Possibilistic quantization Max pooling throug BPT Top-down Bottom-up Part-based model for objects ● ● Automatic determination of parts Efficient BPT analysis. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 92
  93. 93. Open work Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Test in public benchmark. → Instance Search task @ TRECVID 2012 Richer image representations → Amount of BPT leaves adapted to content → Discard uninformative merges & allow multiple options Enrich region descriptors → Introduce interest points → Significance estimation Smarter codebooks → Unsupervised clustering → TF-IDF weights & Stop words Better model for combination of parts representing object Local segmentation of objects from global annotations X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 93
  94. 94. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities 8. Partition Tree Matching Part III 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 94
  95. 95. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Thank you, moltes gràcies ! X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 95

×