Successfully reported this slideshow.
Your SlideShare is downloading. ×

Co-filtering human interaction and object segmentation

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Upcoming SlideShare
IISc Internship Report
IISc Internship Report
Loading in …3
×

Check these out next

1 of 57 Ad

Co-filtering human interaction and object segmentation

Download to read offline

Bachelor thesis by Ferran Cabezas at the INP ENSEEIHT (Toulouse, France). January 2015.

More details:
https://imatge.upc.edu/web/publications/co-filtering-human-interaction-and-object-segmentation

Bachelor thesis by Ferran Cabezas at the INP ENSEEIHT (Toulouse, France). January 2015.

More details:
https://imatge.upc.edu/web/publications/co-filtering-human-interaction-and-object-segmentation

Advertisement
Advertisement

More Related Content

Similar to Co-filtering human interaction and object segmentation (20)

More from Universitat Politècnica de Catalunya (20)

Advertisement

Recently uploaded (20)

Co-filtering human interaction and object segmentation

  1. 1. Co-filtering human interaction and object segmentation Ferran Cabezas Supervised by: Vincent Charvillat Axel Carlier Xavier Giró-i-Nieto Amaia Salvador 1
  2. 2. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 2
  3. 3. Crowdsourcing object segmentation 3
  4. 4. Filtering out bad human interactions Correct human interaction GoalResult of a correct human interaction Result of an incorrect human interaction Incorrect human interaction 4
  5. 5. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 5
  6. 6. Click’n’Cut • Web tool for interactive object segmentation designed for crowdsourcing tasks. A. Carlier, V. Charvillat, A.Salvador, X.Giró-i-Nieto, O. Marques, Click’n’Cut: Crowdsourced Interactive Segmentation with Object Candidates. In CrowdMM’14, 2014 DEMO 6
  7. 7. Data 20 users that have fully realized the Click’n’Cut experiment 100 objects with associated ground truth from the Berkeley-DCU dataset. Testing set 5 images from Pascal VOC 2012 to perform gold standard techniques. Training set Training set 7
  8. 8. How are obtained the masks from the clicks? • Combination of different precomputed binary object candidates . • Foreground map algorithm ? A.Carlier, Combining Content Analysis with Usage Analysis to better understand visual contents, PHD Thesis, 2014. A. Carlier, V. Charvillat, A.Salvador, X.Giró-i-Nieto, O. Marques, Click’n’Cut: Crowdsourced Interactive Segmentation with Object Candidates. In CrowdMM’14, 2014 8
  9. 9. Information of users are not always reliable Bad user interaction Good user interaction 9
  10. 10. First approach - How are separated good from bad user interactions? 4th GS1st GS Error rate Error rate Error rate Error rate Error rate 2nd GS 3rd GS 5th GS Mean error rate • Removing users based on their error rate on the Gold standard images (training set) 10
  11. 11. Removing users based on their error rate Remove users based on an error rate threshold 5GS User20 5GS User18 5GS User19 . . . 5GS User3 5GS User1 5GS User2 Error rate Error rate Error rate Error rate Error rate Error rate 11
  12. 12. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 12
  13. 13. How are evaluated the obtained masks? clicks Object candidate technique Ground truth mask ? ? Foreground map algorithm 13
  14. 14. Jaccard index A ∪ B A ∩ B Measure of similarity between the mask obtained from the Click’n’Cut experiment and the ground truth mask 14
  15. 15. 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates • Removing users • Removing clicks • Removing clicks and users Outline 15
  16. 16. Impact of good and bad users in the resulting mask Image 1 user (good user) Image 12 users (Good users) • A lot of errors can be removed just by discarding bad users Image 20 users 16
  17. 17. Jaccard index= 0.0214 Error rate = 0 Jaccard index= 0.9402 Error rate = 0 Users filtering NO OBVIOUS CORRELATION 17
  18. 18. Jaccard index for each user 4th GS1st GS Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index 2nd GS 3rd GS 5th GS Mean Jaccard index • Better idea of how it is the contribution of the user in the final result 18
  19. 19. Jaccard index for each user 5GS User20 5GS User18 5GS User19 . . . 5GS User3 5GS User1 5GS User2 Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index Remove users based on a Jaccard index threshold 19
  20. 20. Image 100 Jaccard index 100 Image 1 Jaccard index 1 Image 2 Jaccard index 2 Image 3 Jaccard index 3 Image 98 Jaccard index 98 Image 99 Jaccard index 99 MEAN Jaccard index for the test set . . . Maintained users Removed users 20
  21. 21. Results for the test set 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of users Jaccard index by taking different number of users JaccardIndex Users sorted by its ascendent Jaccard index Users sorted by its descendent error rate descendent ascendant 21
  22. 22. 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates • Removing users • Removing clicks • Removing clicks and users Outline 22
  23. 23. Schematic Combination of Object Candidates Image with filtered clicks Obtaining mask Slic Felzenszwalb N-cuts nothin g Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Image with non filtered clicks 23
  24. 24. Schematic Combination of Object Candidates Image with filtered clicks Obtaining mask Slic Felzenszwalb N-cuts nothing Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Image with non filtered clicks 24
  25. 25. Superpixel techniques Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Combination of Object Candidates Slic Felzenszwalb N-cuts nothing Image with filtered clicks Obtaining mask 25
  26. 26. Superpixel techniques • Felzenszwalb • K = 20 • σ = 0,5 • m = 20 • SLIC • Region size = 10 • Regularizer = 0.1• N-cuts 26
  27. 27. Filtering Clicks in a same superpixel Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Combination of Object Candidates Slic Felzenszwalb N-cuts nothing Image with filtered clicks Obtaining mask 27
  28. 28. Filtering Clicks in a same superpixel 1) Total removal of conflict clicks : Discarding all clicks in conflicting superpixels 2) Partial removal of conflict clicks : Discarding the clicks in minority /equality inside conflicting superpixels nothingnothing 28
  29. 29. Results Without applying any technique of filtering clicks 0.14 Techniques of filtering clicks in a same sppxl. Partial removal of conflict clicks Total removal of conflict clicks SLIC 0.2109 0.2412 N-CUTS 0.2735 0.3330 FELZ 0.2104 0.2240 • Jaccard index for all users in the test set 29
  30. 30. 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates • Removing users • Removing clicks • Removing clicks and users Outline 30
  31. 31. Results • Users sorted by its descendent Jaccard index 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard index JaccardIndex Comparing results with partial filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard indexJaccardIndex Comparing results with total filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks Partial filtering Total filtering 31
  32. 32. 3. Treatment of human interaction b) Taking advantage of all human interaction - Foreground map algorithm Outline 32
  33. 33. Foreground map algorithm Set of clicks 50 100 150 200 250 300 350 400 450 50 100 150 200 250 300 50 100 150 200 250 300 350 400 450 50 100 150 200 250 300 Felzenzwalb Superpixel segmentation with k=100 Felzenzwalb Superpixel segmentation with k=300 • Each click have a measure of confidence based on the user error on the 5GS. • Weight superpixel based on clicks 33
  34. 34. Foreground map algorithm • Superpixel combination • Slic: 6 levels • Felzenzwalb: 8 levels . . . . . . R.Vieux, J.Benois, J.Domenger, A.Braquelaire, Segmentation-based multi-class semantic object detection, Multimedia Tools and Applications, 2010 34
  35. 35. Parameters to adjust after the combination • Threshold • Structure element for hole filling ? ? 35
  36. 36. Combining all Felz. and Slic levels Threshold 0.56  Jaccard index = 0.8603 • Felz: k: 10,20,50,100,200,300,400,500 • SLIC: Regions side: 5,10,20,30,40,50 • SE =7 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8891 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the train set 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8603 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the test set 36
  37. 37. Results combining all Felz. and Slic levels Threshold = 0.56 SE = 7 37
  38. 38. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 38
  39. 39. Type of users and their particularities • Painter: Lot of foreground clicks inside the object to segment 39
  40. 40. Type of users and their particularities • Tired: Few clicks per image 40
  41. 41. Type of users and their particularities • Border guards: Most of the bg clicks are in the contour of the image. 41
  42. 42. Type of users and their particularities • Surrounders: Most of the fg clicks are in the contour of the image 42
  43. 43. Type of users and their particularities • Mirrors: Have understood the experiment upside-down 43
  44. 44. Type of users and their particularities • Spammers: Randomly placed foreground clicks over the image. 44
  45. 45. Type of users and their particularities • Experts: Have well-understood the experiment and just made few mistakes 45
  46. 46. Type of users and their particularities • Different pattern: Does not follow the same pattern of clicks in all images 46
  47. 47. Manually categorization • It is done a manually categorization by considering just the 5 gold standard images Users Manually categorization 1 Painter 2 Expert 3 Mirror 4 Expert 5 Border guard 6 Expert 7 Tired 8 Border guard 9 Expert 10 Different pattern 11 Different pattern 12 Expert 13 Expert 14 Expert 15 Expert 16 Expert 17 Tired 18 Surrounder 19 Spammer 20 Expert 47
  48. 48. Manual rules for automatic user categorization Features Painter The mirror The border guard The surrounder The spammer The tired The expert # clicks >150/image - - - - <5/image - fg clicks(%) >95% - <20% >95% >90% - - errors(%) <3% >90% - - >40% <20% - Jaccard index (%) - <10% - - - <80% >80% Contour fg(%) (fg contour clicks/total fg clicks) - - - >80% <80% - - Contour bg(%) (bg contour clicks/total bg clicks) - - >70% - - - - • According to the particularities of each type of user, a set of features and its rules are created: 48
  49. 49. Automatic categorization evaluation for the test set Prediction Painter Mirror Expert Spammer Surrounder Border Guard Tired Diff. Pattern Ground Truth Painter 1 0 0 0 0 0 0 0 Mirror 0 1 0 0 0 0 0 0 Expert 0 0 9 0 0 0 0 1 Spammer 0 0 0 1 0 0 0 0 Surrounder 0 0 0 0 1 0 0 0 Border guard 0 0 0 0 0 1 0 1 Tired 0 0 0 0 0 0 1 1 Diff. pattern 0 0 0 0 0 0 0 2 49
  50. 50. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 50
  51. 51. Conclusions • Jaccard index is a better measure compared to error rate to separate bad users from good ones 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of users Jaccard index by taking different number of users JaccardIndex Users sorted by its ascendent Jaccard index Users sorted by its descendent error rate 51
  52. 52. Conclusions • Better results with partial than with total filtering • Filtering clicks only makes sense when treating with bad users 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard index JaccardIndex Comparing results with partial filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard index JaccardIndex Comparing results with total filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks Partial filtering Total filtering 52
  53. 53. Conclusions • In the foreground map algorithm it is reached the best result by combining Felzenzwalb and Slic superpixel techniques with different levels 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8891 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the train set 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8603 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the test set 53
  54. 54. Conclusions Images from User 11 • It is not possible to automatically categorize users that does not follow the same pattern of clicks in all images 54
  55. 55. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 55
  56. 56. Future work • Study different techniques for filtering clicks in a same superpixel. • Take advantage of the clicks of some users to create a better mask (e.g. Border guard and Surrounder users) • Train classifier for automatic user categorization 56
  57. 57. Questions & Answers 57

×