Part-based Object Retrieval with Binary Partition Trees

1. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Part-based Object Retrieval with Binary Partition Trees Phd thesis dissertation by Xavier Giró-i-Nieto Supervised by Professor Ferran Marqués Acosta (UPC) and Professor Shih-Fu Chang (Columbia) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 1

2. Acknowledgements Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Jury DVMM @ Columbia GPI @ UPC Students @ UPC X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 2

3. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction ← 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 3

4. Problem statement Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Content-Based Image Retrieval X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 4

5. Problem statement Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Semantic gap between user query and machine data. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 5

6. Challenges Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Visual diversity within the same semantic class anchor X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 6

7. Challenges Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Visual diversity within the same instance This anchor X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 7

8. Challenges Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Semantics located at multiple scales News Head TV studio Anchor Button Jacket X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 8

9. Problem statement Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) This thesis focuses in the object retrieval with one example... Query object Target dataset Ranked list X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 9

10. Problem statement Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) ...and multiple examples. Query objects Target dataset Ranked list X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 10

11. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions ← Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 11

12. 2. Hierchical Image Partition Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Object analysis Local scale Sliding windows [Viola-Jones'01] Interest points Image partition SIFT [Lowe'04] SURF [Bay'06] Object Segmentation ? ✘ ✘ X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC ✔ 12

13. 2. Hierchical Image Partition Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Semantics at multiple scales Multi-scale analysis Spatial Pyramid of points [Lazebnik'06] [Bosch'07] Hierarchical Multiple partition segmentations [Ravinovich'07] [Vieux'10] ✔ [Adamek'06] [Gu'09] X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 13

14. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 2. Hierchical Image Partition Adopted Solution: Binary Partition Trees (BPTs) [Garrido, Salembier 2000]. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 14

15. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 2. Hierchical Image Partition Semantic Object “head” Perceptual BPT Split Thesis topic X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 15

16. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation ← 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 16

17. 3. Interactive segmentation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Motivations: Ground truth generation and retrieval interface. Graphical Annotation Tool (GAT) Graphical Object Searcher (GOS) Demo @ http://upseek.upc.edu/gat & http://upseek.upc.edu/gos X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 17

18. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree ← 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 18

19. 4. Object Tree Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Related work: “Definition Hierarchy”, Jaimes and Chang (2001) Thesis approach BPT nodes BPT leaves X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 19

20. 4. Object Tree Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Assumption: All objects are connected Interactive segmentation BPT sub-roots selection X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 20

21. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 4. Object Tree The Object Tree (OT) represents an instance of the object as a hierarchy of regions. User selection of sub-roots on the BPT. Automatic top-down expansion through the BPT X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 21

22. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features ← Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 22

23. 5. Region Features Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) MPEG-7 and many more, “Visual Descriptors (VD)” (2001) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 23

24. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 5. Region Features → Region features in BPT Dominant Colors ➔ Contour Shape ➔ Edge Histogram ➔ NOT Thesis topic X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 24

25. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features ← Part II 6. Codebooks 7. Fusion of Visual Modalities Part III Evaluation 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 25

26. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How to evaluate an object retrieval system ? Annotated dataset + Metrics 1. Image Retrieval 2. Object Detection 3. Object Segmentation 4. Search time X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 26

27. Evaluation (in Part II) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Annotated Dataset ETHZ: Category #1 “Apple logos” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 27

28. Evaluation (in Part II) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Annotated Dataset ETHZ: Category #2 “Bottles” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 28

29. Evaluation (in Part II) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Annotated Dataset ETHZ: Category #3 “Giraffes” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 29

30. Evaluation (in Part II) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Annotated Dataset ETHZ: Category #4 “Mugs” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 30

31. Evaluation (in Part II) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Annotated Dataset ETHZ: Category #5 “Swans” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 31

32. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 1. Image Retrieval (global scale) Only global scale is considered. User Query (local) Ranked list of images (global) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 32

33. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 1. Image Retrieval (global scale) Mean Average Precision (MAP) combines Precision and Recall for a set of queries. Precision Plotted by considering different k's ∣{relevant}∩{retrieved (k )}∣ P (k )= k Recall ∣{relevant }∩{retrieved (k )}∣ R(k )= ∣relevant∣ X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 33

34. 5. Region features Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Image Retrieval (global scale) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 34

35. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 2. Object Detection (local scale-rough) User Query (local) Retrieved boxes in relevant images X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 35

36. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 2. Object Detection (local scale-rough) Pascal criterion requires a minimum of 50% overlapping in the union of detected object box and ground truth box. ∣{correct detections }∣ Detection rate= ∣{relevant }∣ X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 36

37. 5. Region features Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Object Detection (local scale-rough) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 37

38. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 3. Object Segmentation (local scale-precise) User Query (local) Retrieved regions from correct detections X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 38

39. Evaluation Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 3. Object Segmentation (local scale-precise) Jaccard index-J(A,B) compares retrieved region and ground truth mask at the pixel level. ∣A∩B∣ J ( A , B)= ∣A∪B∣ A B Mean Jaccard Index combines the J's obtained by a set of correct detections D. ̄= 1 ∑ J i J ∣D∣ i ∈D X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 39

40. 5. Region features Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Segmentation (local scale-precise) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 40

41. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks ← 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 41

42. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 6. Codebooks Visual Word (VW): Vector quantization of observation (point, region...) onto a codebook [Sivic, Zissermann 2003]. Regions Codebook Code Region Visual Words (a1,a2,a3,a4) (b1,b2,b3,b4) (c1,c2,c3,c4) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 42

43. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Code Regions (CR) define the basis of the Parts Space. “Regions described by their sub-parts” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 43

44. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (a) Hard quantization SIFT [Sivic 2003] Regions Codebook Visual Words ✘ (1,0,0,0) ? (1,0,0,0) (0,1,0,0) ? X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC ✘ 44

45. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (b) Probabilistic quantization [Alpaydin 1998] Regions Codebook Visual Words (0.25,0.25,0.25,0.25) ✘ (1,0,0,0) ✘ (0.25,0.25,0.25,0.25) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 45

46. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (c) Possibilistic quantization [Bezdek 1995] Regions Codebook Visual Words (0,0,0,0) ✔ (1,0,0,0) ✘ (0,0,0,0) Similarity to CRs X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 46

47. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How to define the Parts Space with a given codebook ? Possibilistic quantization alone characterizes regions, but not their sub-parts. Region-based Possibilistic X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 47

48. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How to define the Parts Space with a given codebook ? Bottom-up Max Pooling through BPT ✔ [Boureau'10] Fast Extraction (max) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 48

49. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) → Hierarchy of Bags of Regions (HBoR) HBoR BPT is exploited to describe sub-trees. Fast Extraction (max) Efficient Analysis (resilency to splits) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 49

50. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How to generate a codebook ? (1/2) Train set BPT-based decomposition Object parts K-Means Clustering Codebook CR1 CR2 CR3 CR4 X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 50

51. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How to generate a codebook ? (2/2) a) Foreground and background regions - “generic” b) Only object regions - “frgd” X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 51

52. 6. Codebooks Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 1. Image Retrieval (global scale) - Texture HBoR HBoR features outstand region-based for shape & texture (not dominant colors). Little differences between generic vs frgd codebooks (5 classes in ETHZ). Codebook size X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 52

53. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features ← Part II 6. Codebooks 7. Fusion of Visual Modalities ← Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 53

54. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 5. Region Features + 7. Fusion Problem: Similarity scores between descriptors are not comparable. Solution: d color d shape d texture Normalization (z-scores, SVM, w-scores) x ̄color x ̄ shape x ̄texture Fusion (average, product, max/min) x ̄ X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 54

55. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks ← 7. Fusion of Visual Modalities ← Part III 8. Partition Tree Matching 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 55

56. 6. Codebooks + 7. Fusion Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) When working with codebooks, when to apply fusion ? (a) Fusion after quantization - “nofused” Query region [ ] color shape texture Target region [ ] color shape texture Separate codebooks Quantization [ ] [ ] VW color VW shape VW texture VW color VW shape VW texture Cosine distance [ ] x ̄color x ̄ shape x ̄ texture Fusion x X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 56

57. 6. Codebooks + 7. Fusion Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) When working with codebooks, when to apply fusion ? (b) Fusion during quantization - “fused” Query region [ ] color shape texture Target region [ ] color shape texture Fused codebook Quantization [ VW ] [ VW ] Cosine distance x X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 57

58. 6. Codebooks + 7. Fusion Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) When working with codebooks, when to apply fusion ? 1. Image Retrieval (global scale) z-scores w-scores W-scores outperform z-scores when working with HBoR. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 58

59. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching ← 9. Part-based Object Model ← Evaluation 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 59

60. Evaluation (in Part III) Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) CCMA News dataset. Topology diversity on 11 anchors objects. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 60

61. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities Part III 8. Partition Tree Matching ← 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 61

62. 8. Partition Tree Matching Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Reminder: The Object Retrieval Problem (single instance) Query object Target dataset Ranked list X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 62

63. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) 8. Partition Tree Matching Object Tree (OT) Target BPT Q0 Q1 ? Q2 Q3 Q4 Challenges: (a) Distance for a given OT-BPT correspondence (b) Matching algorithm X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 63

64. 8. Partition Tree Matching: Distance Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Proposal (ad-hoc): Weighted sum of part matches N d =∑ wi d i , where w i=i i= 0 ∏ Q   j =i⋅w Parent Q  j ∈ Ancestors i i area(i) 1 αi = i i max (1,∣C ∣) area(P ) Children of part i Example: Q0 Q1 Q2 Parents of part i α0=1 /2=w 0 α1=area(Q 1)/area (Q 0) α 2=area(Q 2 )/area(Q0 ) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 64

65. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Two proposals: (a) Top-down Matching (b) Bottom-Up Matching Object Tree (OT) Target BPT Q0 Q1 ? Q2 Q3 Q4 X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 65

66. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (a) Top-down Matching Assumption: All OT nodes are contained in a single sub-BPT. -c +c n mo om om mo n Sets a baseline (b) Bottom-Up Matching Assumption: OT nodes are spread through different subBPTs. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 66

67. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (b) Top-down QT Matching Assumption: The best match for the OT is contained in a single sub-tree of the target BPT. OT Target BPT Q0 1. Set a match for Q0 Q1 Q2 Q0 2. Find matches for Q1 and Q2 Q1 Q2 X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 67

68. 8. Partition Tree Matching Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) What is the impact of decomposing the query into parts ? Image Retrieval (global scale) MPEG-7 Region Top-down match Retrieval gain until the “correct” amount of parts (3). X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 68

69. 8. Partition Tree Matching Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) What is the impact of decomposing the query into parts ? Search time MPEG-7 Region Top-down match Exponential growth of search time with the amount of parts. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 69

70. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (a) Top-down Matching Assumption: All OT nodes are contained in a single sub-BPT. -c +c n mo om om mo n Sets a baseline (b) Bottom-Up Matching Assumption: OT nodes are spread through different subBPTs. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 70

71. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (c) Bottom-UP QT Matching Q0 1. Set a match for Q1 & Q2 Q1 2. Assess the union of candidates for Q1 and Q2 to match Q0 Q2 This region is not part of the target BPT Q0 Q1 Q2 Weaker assumption that in the top-down case: → OT leaves are among the nodes of the target BPT X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 71

72. 8. Partition Tree Matching: Algorithm Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) (c) Bottom-UP QT Matching How to rapidly describe the merged region generated during search time ? HBoR Object Tree (1, 1, 1, 1) Q0 Q1 Q2 → Approximate HBoR satisfies this requirement. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 72

73. 8. Partition Tree Matching Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) What is the impact of introducing the bottom-up match ? HBoR Bottom-up gain Region VW A clear gain, the object splits in the target BPTs are solved. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 73

74. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities 8. Partition Tree Matching Part III 9. Part-based Object Model ← 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 74

75. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Reminder: The Object Retrieval Problem (multiple instances) Query objects Target dataset Ranked list X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 75

76. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Model-based retrieval Query objects Model Training Target image Retrieved object Test X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 76

77. 9. Part-based Object Model Training Fusion classifier Inclusion classifier #P ar ts Test Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Parts definition Inclusion classifier #P ar ts Detector classifier Detector classifier Fusion classifier Contextual filtering X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 77

79. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Parts definition Unsupervised clustering with Quality Threshold [Heyer 1999]. Two parameters: - Mininum amount of elements in cluster - Maximum radius of a cluster X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 79

81. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Inclusion classifier (training) Positive: Parts and their ancestors Negative: Parts' children and positive node's siblings. HBoR X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 81

83. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Inclusion classifier (test) Efficient top-down exploration of the target BPT. HBoR Not explored (efficiency) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 83

85. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Detector classifier (training) Positive: Parts. Negative: Random sampling (balanced set). Region-based Possibilistic X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 85

87. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Fusion classifier (training) Positive training samples represent valid combinations of parts. Negative training samples represent invalid combinations. Parts definition Type 1 Parts of object i (1, 1, 0, 0) Type 2 Type 3 Type 4 Parts of object j (0, 1, 1, 1) X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 87

89. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) How important is a correct estimation of the types of parts ? Image Retrieval Max radius for cluster Min # elems in cluster CodeBook size The performance is dependent from the Quality Threshold parameters. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 89

90. 9. Part-based Object Model Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) What is the impact of codebook size in terms of time ? Training time Search time Search time does not grow with codebook size (Efficient exploration ??). X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 90

91. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities 8. Partition Tree Matching Part III 9. Part-based Object Model 10.Conclusions ← X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 91

92. Conclusions Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) BPT presents several opportunities for local analysis: → Interactive segmentation → Semantic analysis Three main contributions in this thesis: Hierarchy of Bags of Regions: Contribution #1 Contribution #2 ● ● Object Tree - BPT matching ● ● Contribution #3 Possibilistic quantization Max pooling throug BPT Top-down Bottom-up Part-based model for objects ● ● Automatic determination of parts Efficient BPT analysis. X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 92

93. Open work Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Test in public benchmark. → Instance Search task @ TRECVID 2012 Richer image representations → Amount of BPT leaves adapted to content → Discard uninformative merges & allow multiple options Enrich region descriptors → Introduce interest points → Significance estimation Smarter codebooks → Unsupervised clustering → TF-IDF weights & Stop words Better model for combination of parts representing object Local segmentation of objects from global annotations X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 93

94. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Outline 1. Introduction 2. Hierarchical Image Partitions Part I 3. Interactive segmentation 4. Object Tree 5. Region Features Part II 6. Codebooks 7. Fusion of Visual Modalities 8. Partition Tree Matching Part III 9. Part-based Object Model 10.Conclusions X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 94

95. Grup de Processament de la Imatge Universitat Politècnica de Catalunya (UPC) Thank you, moltes gràcies ! X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 95