SlideShare a Scribd company logo
1 of 57
Download to read offline
Co-filtering human interaction
and object segmentation
Ferran Cabezas
Supervised by:
Vincent Charvillat
Axel Carlier
Xavier Giró-i-Nieto
Amaia Salvador
1
1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
2
Crowdsourcing object segmentation
3
Filtering out bad human interactions
Correct human interaction
GoalResult of a correct human interaction Result of an incorrect human interaction
Incorrect human interaction
4
1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
5
Click’n’Cut
• Web tool for interactive object segmentation designed for crowdsourcing
tasks.
A. Carlier, V. Charvillat, A.Salvador, X.Giró-i-Nieto, O. Marques, Click’n’Cut: Crowdsourced Interactive Segmentation with Object
Candidates. In CrowdMM’14, 2014
DEMO
6
Data
20 users that have
fully realized the
Click’n’Cut experiment
100 objects with
associated ground
truth from the
Berkeley-DCU dataset.
Testing set
5 images from Pascal VOC
2012 to perform gold
standard techniques.
Training set
Training set
7
How are obtained the masks from the clicks?
• Combination of different precomputed
binary object candidates .
• Foreground map algorithm
?
A.Carlier, Combining Content Analysis with Usage Analysis to better understand visual
contents, PHD Thesis, 2014.
A. Carlier, V. Charvillat, A.Salvador, X.Giró-i-Nieto, O. Marques, Click’n’Cut:
Crowdsourced Interactive Segmentation with Object Candidates. In
CrowdMM’14, 2014
8
Information of users are not always reliable
Bad user interaction Good user interaction
9
First approach - How are separated good from bad
user interactions?
4th GS1st GS
Error rate Error rate Error rate Error rate Error rate
2nd GS 3rd GS 5th GS
Mean error rate
• Removing users based on their error rate on the Gold standard images (training set)
10
Removing users based on their error rate
Remove users based on an error rate threshold
5GS
User20
5GS
User18
5GS
User19
. . .
5GS
User3
5GS
User1
5GS
User2
Error rate Error rate Error rate Error rate Error rate Error rate
11
1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
12
How are evaluated the obtained masks?
clicks
Object
candidate
technique
Ground truth mask
?
?
Foreground
map algorithm
13
Jaccard index
A ∪ B
A ∩ B
Measure of similarity between the mask obtained from the Click’n’Cut experiment and the ground
truth mask
14
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
• Removing users
• Removing clicks
• Removing clicks and users
Outline
15
Impact of good and bad users in the resulting mask
Image
1 user (good user)
Image
12 users (Good users)
• A lot of errors can be removed just by discarding bad users
Image
20 users
16
Jaccard index= 0.0214
Error rate = 0
Jaccard index= 0.9402
Error rate = 0
Users filtering
NO OBVIOUS CORRELATION
17
Jaccard index for each user
4th GS1st GS
Jaccard
index
Jaccard
index
Jaccard
index
Jaccard
index
Jaccard
index
2nd GS 3rd GS 5th GS
Mean Jaccard index
• Better idea of how it is the contribution of the user in the final result
18
Jaccard index for each user
5GS
User20
5GS
User18
5GS
User19
. . .
5GS
User3
5GS
User1
5GS
User2
Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index
Remove users based on a Jaccard index threshold
19
Image 100
Jaccard index 100
Image 1
Jaccard index 1
Image 2
Jaccard index 2
Image 3
Jaccard index 3
Image 98
Jaccard index 98
Image 99
Jaccard index 99
MEAN
Jaccard index for the test set
. . .
Maintained users
Removed users
20
Results for the test set
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of users
Jaccard index by taking different number of users
JaccardIndex
Users sorted by its ascendent Jaccard index
Users sorted by its descendent error rate
descendent
ascendant
21
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
• Removing users
• Removing clicks
• Removing clicks and users
Outline
22
Schematic
Combination of
Object
Candidates
Image with filtered clicks
Obtaining mask
Slic
Felzenszwalb
N-cuts
nothin
g
Three different
techniques for over-
segment an image
Two techniques for discarding
the clicks in a same superpixel
Image with non filtered clicks
23
Schematic
Combination of
Object
Candidates
Image with filtered clicks
Obtaining mask
Slic
Felzenszwalb
N-cuts
nothing
Three different
techniques for over-
segment an image
Two techniques for discarding
the clicks in a same superpixel
Image with non filtered clicks
24
Superpixel techniques
Three different
techniques for over-
segment an image
Two techniques for discarding
the clicks in a same superpixel
Combination of
Object
Candidates
Slic
Felzenszwalb
N-cuts
nothing
Image with filtered clicks
Obtaining mask
25
Superpixel techniques
• Felzenszwalb
• K = 20
• σ = 0,5
• m = 20
• SLIC
• Region size = 10
• Regularizer = 0.1• N-cuts
26
Filtering Clicks in a same superpixel
Three different
techniques for over-
segment an image
Two techniques for discarding
the clicks in a same superpixel
Combination of
Object
Candidates
Slic
Felzenszwalb
N-cuts
nothing
Image with filtered clicks
Obtaining mask
27
Filtering Clicks in a same superpixel
1) Total removal of conflict clicks :
Discarding all clicks in conflicting
superpixels
2) Partial removal of conflict clicks :
Discarding the clicks in minority
/equality inside conflicting
superpixels
nothingnothing
28
Results
Without applying any
technique of filtering
clicks
0.14
Techniques of
filtering clicks in a
same sppxl.
Partial removal of
conflict clicks
Total removal of
conflict clicks
SLIC 0.2109 0.2412
N-CUTS 0.2735 0.3330
FELZ 0.2104 0.2240
• Jaccard index for all users in the test set
29
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
• Removing users
• Removing clicks
• Removing clicks and users
Outline
30
Results
• Users sorted by its descendent Jaccard index
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of Users sorted by its descended Jaccard index
JaccardIndex
Comparing results with partial filtering and without filtering
Felz. sppxl. technique
Ncuts spxxl. technique
SLIC spxxl. technique
With no filtering clicks
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of Users sorted by its descended Jaccard indexJaccardIndex
Comparing results with total filtering and without filtering
Felz. sppxl. technique
Ncuts spxxl. technique
SLIC spxxl. technique
With no filtering clicks
Partial filtering Total filtering
31
3. Treatment of human interaction
b) Taking advantage of all human interaction - Foreground map algorithm
Outline
32
Foreground map algorithm
Set of clicks
50 100 150 200 250 300 350 400 450
50
100
150
200
250
300
50 100 150 200 250 300 350 400 450
50
100
150
200
250
300
Felzenzwalb
Superpixel
segmentation
with k=100
Felzenzwalb
Superpixel
segmentation
with k=300
• Each click have a measure of confidence
based on the user error on the 5GS.
• Weight superpixel based on clicks
33
Foreground map algorithm
• Superpixel combination
• Slic: 6 levels
• Felzenzwalb: 8 levels
. . . . . .
R.Vieux, J.Benois, J.Domenger, A.Braquelaire,
Segmentation-based multi-class semantic object
detection, Multimedia Tools and Applications, 2010 34
Parameters to adjust after the combination
• Threshold
• Structure element for hole filling
?
?
35
Combining all Felz. and Slic levels
Threshold 0.56  Jaccard index = 0.8603
• Felz: k: 10,20,50,100,200,300,400,500
• SLIC: Regions side: 5,10,20,30,40,50
• SE =7
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1 X: 0.56
Y: 0.8891
Threshold
JaccardIndex
Combining Slic and Felzenzwalb superpixels techniques in the train set
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
X: 0.56
Y: 0.8603
Threshold
JaccardIndex
Combining Slic and Felzenzwalb superpixels techniques in the test set
36
Results combining all Felz. and Slic levels
Threshold = 0.56
SE = 7
37
1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
38
Type of users and their particularities
• Painter: Lot of foreground clicks inside the object to segment
39
Type of users and their particularities
• Tired: Few clicks per image
40
Type of users and their particularities
• Border guards: Most of the bg clicks are in the contour of the image.
41
Type of users and their particularities
• Surrounders: Most of the fg clicks are in the contour of the image
42
Type of users and their particularities
• Mirrors: Have understood the experiment upside-down
43
Type of users and their particularities
• Spammers: Randomly placed foreground clicks over the image.
44
Type of users and their particularities
• Experts: Have well-understood the experiment and just made few
mistakes
45
Type of users and their particularities
• Different pattern: Does not follow the same pattern of clicks in all images
46
Manually categorization
• It is done a manually
categorization by considering just
the 5 gold standard images
Users Manually categorization
1 Painter
2 Expert
3 Mirror
4 Expert
5 Border guard
6 Expert
7 Tired
8 Border guard
9 Expert
10 Different pattern
11 Different pattern
12 Expert
13 Expert
14 Expert
15 Expert
16 Expert
17 Tired
18 Surrounder
19 Spammer
20 Expert
47
Manual rules for automatic user categorization
Features Painter The
mirror
The border
guard
The
surrounder
The
spammer
The tired The expert
# clicks >150/image - - - - <5/image -
fg clicks(%) >95% - <20% >95% >90% - -
errors(%)
<3% >90% - - >40% <20% -
Jaccard index (%) - <10% - - - <80% >80%
Contour fg(%)
(fg contour clicks/total fg
clicks)
- - - >80% <80% - -
Contour bg(%)
(bg contour clicks/total bg
clicks)
- - >70% - - - -
• According to the particularities of each type of user, a set of features and its rules are created:
48
Automatic categorization evaluation for the test set
Prediction
Painter Mirror Expert Spammer Surrounder Border Guard Tired Diff. Pattern
Ground Truth
Painter 1 0 0 0 0 0 0 0
Mirror 0 1 0 0 0 0 0 0
Expert 0 0 9 0 0 0 0 1
Spammer 0 0 0 1 0 0 0 0
Surrounder 0 0 0 0 1 0 0 0
Border guard 0 0 0 0 0 1 0 1
Tired 0 0 0 0 0 0 1 1
Diff. pattern 0 0 0 0 0 0 0 2
49
1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
50
Conclusions
• Jaccard index is a better measure compared to error rate to separate bad
users from good ones
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of users
Jaccard index by taking different number of users
JaccardIndex
Users sorted by its ascendent Jaccard index
Users sorted by its descendent error rate
51
Conclusions
• Better results with partial than with total filtering
• Filtering clicks only makes sense when treating with bad users
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of Users sorted by its descended Jaccard index
JaccardIndex
Comparing results with partial filtering and without filtering
Felz. sppxl. technique
Ncuts spxxl. technique
SLIC spxxl. technique
With no filtering clicks
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of Users sorted by its descended Jaccard index
JaccardIndex
Comparing results with total filtering and without filtering
Felz. sppxl. technique
Ncuts spxxl. technique
SLIC spxxl. technique
With no filtering clicks
Partial filtering
Total filtering
52
Conclusions
• In the foreground map algorithm it is reached the best result by
combining Felzenzwalb and Slic superpixel techniques with different levels
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1 X: 0.56
Y: 0.8891
Threshold
JaccardIndex
Combining Slic and Felzenzwalb superpixels techniques in the train set
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
X: 0.56
Y: 0.8603
Threshold
JaccardIndex
Combining Slic and Felzenzwalb superpixels techniques in the test set
53
Conclusions
Images from User 11
• It is not possible to automatically categorize users that does not
follow the same pattern of clicks in all images
54
1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
55
Future work
• Study different techniques for filtering clicks in a same superpixel.
• Take advantage of the clicks of some users to create a better mask
(e.g. Border guard and Surrounder users)
• Train classifier for automatic user categorization
56
Questions & Answers
57

More Related Content

Similar to Co-filtering human interaction and object segmentation

Paper id 312201522
Paper id 312201522Paper id 312201522
Paper id 312201522IJRAT
 
Face Recognition Based Intelligent Door Control System
Face Recognition Based Intelligent Door Control SystemFace Recognition Based Intelligent Door Control System
Face Recognition Based Intelligent Door Control Systemijtsrd
 
Rapid object detection using boosted cascade of simple features
Rapid object detection using boosted  cascade of simple featuresRapid object detection using boosted  cascade of simple features
Rapid object detection using boosted cascade of simple featuresHirantha Pradeep
 
IRJET - A Systematic Observation in Digital Image Forgery Detection using MATLAB
IRJET - A Systematic Observation in Digital Image Forgery Detection using MATLABIRJET - A Systematic Observation in Digital Image Forgery Detection using MATLAB
IRJET - A Systematic Observation in Digital Image Forgery Detection using MATLABIRJET Journal
 
Face Recognition & Detection Using Image Processing
Face Recognition & Detection Using Image ProcessingFace Recognition & Detection Using Image Processing
Face Recognition & Detection Using Image Processingpaperpublications3
 
FACE MASK DETECTION USING MACHINE LEARNING AND IMAGE PROCESSING
FACE MASK DETECTION USING MACHINE LEARNING AND IMAGE PROCESSINGFACE MASK DETECTION USING MACHINE LEARNING AND IMAGE PROCESSING
FACE MASK DETECTION USING MACHINE LEARNING AND IMAGE PROCESSINGIRJET Journal
 
Road signs detection using voila jone's algorithm with the help of opencv
Road signs detection using voila jone's algorithm with the help of opencvRoad signs detection using voila jone's algorithm with the help of opencv
Road signs detection using voila jone's algorithm with the help of opencvMohdSalim34
 
International Journal of Engineering and Science Invention (IJESI)
International Journal of Engineering and Science Invention (IJESI)International Journal of Engineering and Science Invention (IJESI)
International Journal of Engineering and Science Invention (IJESI)inventionjournals
 
Recent advances in deep recommender systems
Recent advances in deep recommender systemsRecent advances in deep recommender systems
Recent advances in deep recommender systemsNAVER Engineering
 
Face Recognition & Detection Using Image Processing
Face Recognition & Detection Using Image ProcessingFace Recognition & Detection Using Image Processing
Face Recognition & Detection Using Image Processingpaperpublications3
 
TechnicalBackgroundOverview
TechnicalBackgroundOverviewTechnicalBackgroundOverview
TechnicalBackgroundOverviewMotaz El-Saban
 
IRJET- Generating 3D Models Using 3D Generative Adversarial Network
IRJET- Generating 3D Models Using 3D Generative Adversarial NetworkIRJET- Generating 3D Models Using 3D Generative Adversarial Network
IRJET- Generating 3D Models Using 3D Generative Adversarial NetworkIRJET Journal
 
Tracking Robustness and Green View Index Estimation of Augmented and Diminish...
Tracking Robustness and Green View Index Estimation of Augmented and Diminish...Tracking Robustness and Green View Index Estimation of Augmented and Diminish...
Tracking Robustness and Green View Index Estimation of Augmented and Diminish...Tomohiro Fukuda
 
Image Forgery Detection Methods- A Review
Image Forgery Detection Methods- A ReviewImage Forgery Detection Methods- A Review
Image Forgery Detection Methods- A ReviewIRJET Journal
 
IRJET - Hand Gesture Recognition to Perform System Operations
IRJET -  	  Hand Gesture Recognition to Perform System OperationsIRJET -  	  Hand Gesture Recognition to Perform System Operations
IRJET - Hand Gesture Recognition to Perform System OperationsIRJET Journal
 
EyeGrip: Detecting Targets in a Series of Uni-directional Moving Objects Usin...
EyeGrip: Detecting Targets in a Series of Uni-directional Moving Objects Usin...EyeGrip: Detecting Targets in a Series of Uni-directional Moving Objects Usin...
EyeGrip: Detecting Targets in a Series of Uni-directional Moving Objects Usin...Diako Mardanbegi
 

Similar to Co-filtering human interaction and object segmentation (20)

CrowDM system
CrowDM systemCrowDM system
CrowDM system
 
Paper id 312201522
Paper id 312201522Paper id 312201522
Paper id 312201522
 
Face Recognition Based Intelligent Door Control System
Face Recognition Based Intelligent Door Control SystemFace Recognition Based Intelligent Door Control System
Face Recognition Based Intelligent Door Control System
 
[DL輪読会]ClearGrasp
[DL輪読会]ClearGrasp[DL輪読会]ClearGrasp
[DL輪読会]ClearGrasp
 
Rapid object detection using boosted cascade of simple features
Rapid object detection using boosted  cascade of simple featuresRapid object detection using boosted  cascade of simple features
Rapid object detection using boosted cascade of simple features
 
IRJET - A Systematic Observation in Digital Image Forgery Detection using MATLAB
IRJET - A Systematic Observation in Digital Image Forgery Detection using MATLABIRJET - A Systematic Observation in Digital Image Forgery Detection using MATLAB
IRJET - A Systematic Observation in Digital Image Forgery Detection using MATLAB
 
Face Recognition & Detection Using Image Processing
Face Recognition & Detection Using Image ProcessingFace Recognition & Detection Using Image Processing
Face Recognition & Detection Using Image Processing
 
FACE MASK DETECTION USING MACHINE LEARNING AND IMAGE PROCESSING
FACE MASK DETECTION USING MACHINE LEARNING AND IMAGE PROCESSINGFACE MASK DETECTION USING MACHINE LEARNING AND IMAGE PROCESSING
FACE MASK DETECTION USING MACHINE LEARNING AND IMAGE PROCESSING
 
Road signs detection using voila jone's algorithm with the help of opencv
Road signs detection using voila jone's algorithm with the help of opencvRoad signs detection using voila jone's algorithm with the help of opencv
Road signs detection using voila jone's algorithm with the help of opencv
 
final ppt
final pptfinal ppt
final ppt
 
International Journal of Engineering and Science Invention (IJESI)
International Journal of Engineering and Science Invention (IJESI)International Journal of Engineering and Science Invention (IJESI)
International Journal of Engineering and Science Invention (IJESI)
 
Recent advances in deep recommender systems
Recent advances in deep recommender systemsRecent advances in deep recommender systems
Recent advances in deep recommender systems
 
Face Recognition & Detection Using Image Processing
Face Recognition & Detection Using Image ProcessingFace Recognition & Detection Using Image Processing
Face Recognition & Detection Using Image Processing
 
TechnicalBackgroundOverview
TechnicalBackgroundOverviewTechnicalBackgroundOverview
TechnicalBackgroundOverview
 
IRJET- Generating 3D Models Using 3D Generative Adversarial Network
IRJET- Generating 3D Models Using 3D Generative Adversarial NetworkIRJET- Generating 3D Models Using 3D Generative Adversarial Network
IRJET- Generating 3D Models Using 3D Generative Adversarial Network
 
Tracking Robustness and Green View Index Estimation of Augmented and Diminish...
Tracking Robustness and Green View Index Estimation of Augmented and Diminish...Tracking Robustness and Green View Index Estimation of Augmented and Diminish...
Tracking Robustness and Green View Index Estimation of Augmented and Diminish...
 
Clickstream ppt copy
Clickstream ppt   copyClickstream ppt   copy
Clickstream ppt copy
 
Image Forgery Detection Methods- A Review
Image Forgery Detection Methods- A ReviewImage Forgery Detection Methods- A Review
Image Forgery Detection Methods- A Review
 
IRJET - Hand Gesture Recognition to Perform System Operations
IRJET -  	  Hand Gesture Recognition to Perform System OperationsIRJET -  	  Hand Gesture Recognition to Perform System Operations
IRJET - Hand Gesture Recognition to Perform System Operations
 
EyeGrip: Detecting Targets in a Series of Uni-directional Moving Objects Usin...
EyeGrip: Detecting Targets in a Series of Uni-directional Moving Objects Usin...EyeGrip: Detecting Targets in a Series of Uni-directional Moving Objects Usin...
EyeGrip: Detecting Targets in a Series of Uni-directional Moving Objects Usin...
 

More from Universitat Politècnica de Catalunya

The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...Universitat Politècnica de Catalunya
 
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-NietoTowards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-NietoUniversitat Politècnica de Catalunya
 
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...Universitat Politècnica de Catalunya
 
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in VideosGeneration of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in VideosUniversitat Politècnica de Catalunya
 
Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...Universitat Politècnica de Catalunya
 
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020Universitat Politècnica de Catalunya
 
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...Universitat Politècnica de Catalunya
 
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020Universitat Politècnica de Catalunya
 
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...Universitat Politècnica de Catalunya
 
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020Universitat Politècnica de Catalunya
 
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)Universitat Politècnica de Catalunya
 
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...Universitat Politècnica de Catalunya
 
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020Universitat Politècnica de Catalunya
 

More from Universitat Politècnica de Catalunya (20)

Deep Generative Learning for All
Deep Generative Learning for AllDeep Generative Learning for All
Deep Generative Learning for All
 
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
 
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-NietoTowards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
 
The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021
 
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
 
Open challenges in sign language translation and production
Open challenges in sign language translation and productionOpen challenges in sign language translation and production
Open challenges in sign language translation and production
 
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in VideosGeneration of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
 
Discovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in MinecraftDiscovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in Minecraft
 
Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...
 
Intepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural NetworksIntepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural Networks
 
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
 
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
 
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
 
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
 
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
 
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
 
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
 
Curriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object SegmentationCurriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object Segmentation
 
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
 
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
 

Recently uploaded

[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Scott Andery
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 

Recently uploaded (20)

[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 

Co-filtering human interaction and object segmentation

  • 1. Co-filtering human interaction and object segmentation Ferran Cabezas Supervised by: Vincent Charvillat Axel Carlier Xavier Giró-i-Nieto Amaia Salvador 1
  • 2. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 2
  • 4. Filtering out bad human interactions Correct human interaction GoalResult of a correct human interaction Result of an incorrect human interaction Incorrect human interaction 4
  • 5. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 5
  • 6. Click’n’Cut • Web tool for interactive object segmentation designed for crowdsourcing tasks. A. Carlier, V. Charvillat, A.Salvador, X.Giró-i-Nieto, O. Marques, Click’n’Cut: Crowdsourced Interactive Segmentation with Object Candidates. In CrowdMM’14, 2014 DEMO 6
  • 7. Data 20 users that have fully realized the Click’n’Cut experiment 100 objects with associated ground truth from the Berkeley-DCU dataset. Testing set 5 images from Pascal VOC 2012 to perform gold standard techniques. Training set Training set 7
  • 8. How are obtained the masks from the clicks? • Combination of different precomputed binary object candidates . • Foreground map algorithm ? A.Carlier, Combining Content Analysis with Usage Analysis to better understand visual contents, PHD Thesis, 2014. A. Carlier, V. Charvillat, A.Salvador, X.Giró-i-Nieto, O. Marques, Click’n’Cut: Crowdsourced Interactive Segmentation with Object Candidates. In CrowdMM’14, 2014 8
  • 9. Information of users are not always reliable Bad user interaction Good user interaction 9
  • 10. First approach - How are separated good from bad user interactions? 4th GS1st GS Error rate Error rate Error rate Error rate Error rate 2nd GS 3rd GS 5th GS Mean error rate • Removing users based on their error rate on the Gold standard images (training set) 10
  • 11. Removing users based on their error rate Remove users based on an error rate threshold 5GS User20 5GS User18 5GS User19 . . . 5GS User3 5GS User1 5GS User2 Error rate Error rate Error rate Error rate Error rate Error rate 11
  • 12. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 12
  • 13. How are evaluated the obtained masks? clicks Object candidate technique Ground truth mask ? ? Foreground map algorithm 13
  • 14. Jaccard index A ∪ B A ∩ B Measure of similarity between the mask obtained from the Click’n’Cut experiment and the ground truth mask 14
  • 15. 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates • Removing users • Removing clicks • Removing clicks and users Outline 15
  • 16. Impact of good and bad users in the resulting mask Image 1 user (good user) Image 12 users (Good users) • A lot of errors can be removed just by discarding bad users Image 20 users 16
  • 17. Jaccard index= 0.0214 Error rate = 0 Jaccard index= 0.9402 Error rate = 0 Users filtering NO OBVIOUS CORRELATION 17
  • 18. Jaccard index for each user 4th GS1st GS Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index 2nd GS 3rd GS 5th GS Mean Jaccard index • Better idea of how it is the contribution of the user in the final result 18
  • 19. Jaccard index for each user 5GS User20 5GS User18 5GS User19 . . . 5GS User3 5GS User1 5GS User2 Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index Remove users based on a Jaccard index threshold 19
  • 20. Image 100 Jaccard index 100 Image 1 Jaccard index 1 Image 2 Jaccard index 2 Image 3 Jaccard index 3 Image 98 Jaccard index 98 Image 99 Jaccard index 99 MEAN Jaccard index for the test set . . . Maintained users Removed users 20
  • 21. Results for the test set 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of users Jaccard index by taking different number of users JaccardIndex Users sorted by its ascendent Jaccard index Users sorted by its descendent error rate descendent ascendant 21
  • 22. 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates • Removing users • Removing clicks • Removing clicks and users Outline 22
  • 23. Schematic Combination of Object Candidates Image with filtered clicks Obtaining mask Slic Felzenszwalb N-cuts nothin g Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Image with non filtered clicks 23
  • 24. Schematic Combination of Object Candidates Image with filtered clicks Obtaining mask Slic Felzenszwalb N-cuts nothing Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Image with non filtered clicks 24
  • 25. Superpixel techniques Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Combination of Object Candidates Slic Felzenszwalb N-cuts nothing Image with filtered clicks Obtaining mask 25
  • 26. Superpixel techniques • Felzenszwalb • K = 20 • σ = 0,5 • m = 20 • SLIC • Region size = 10 • Regularizer = 0.1• N-cuts 26
  • 27. Filtering Clicks in a same superpixel Three different techniques for over- segment an image Two techniques for discarding the clicks in a same superpixel Combination of Object Candidates Slic Felzenszwalb N-cuts nothing Image with filtered clicks Obtaining mask 27
  • 28. Filtering Clicks in a same superpixel 1) Total removal of conflict clicks : Discarding all clicks in conflicting superpixels 2) Partial removal of conflict clicks : Discarding the clicks in minority /equality inside conflicting superpixels nothingnothing 28
  • 29. Results Without applying any technique of filtering clicks 0.14 Techniques of filtering clicks in a same sppxl. Partial removal of conflict clicks Total removal of conflict clicks SLIC 0.2109 0.2412 N-CUTS 0.2735 0.3330 FELZ 0.2104 0.2240 • Jaccard index for all users in the test set 29
  • 30. 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates • Removing users • Removing clicks • Removing clicks and users Outline 30
  • 31. Results • Users sorted by its descendent Jaccard index 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard index JaccardIndex Comparing results with partial filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard indexJaccardIndex Comparing results with total filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks Partial filtering Total filtering 31
  • 32. 3. Treatment of human interaction b) Taking advantage of all human interaction - Foreground map algorithm Outline 32
  • 33. Foreground map algorithm Set of clicks 50 100 150 200 250 300 350 400 450 50 100 150 200 250 300 50 100 150 200 250 300 350 400 450 50 100 150 200 250 300 Felzenzwalb Superpixel segmentation with k=100 Felzenzwalb Superpixel segmentation with k=300 • Each click have a measure of confidence based on the user error on the 5GS. • Weight superpixel based on clicks 33
  • 34. Foreground map algorithm • Superpixel combination • Slic: 6 levels • Felzenzwalb: 8 levels . . . . . . R.Vieux, J.Benois, J.Domenger, A.Braquelaire, Segmentation-based multi-class semantic object detection, Multimedia Tools and Applications, 2010 34
  • 35. Parameters to adjust after the combination • Threshold • Structure element for hole filling ? ? 35
  • 36. Combining all Felz. and Slic levels Threshold 0.56  Jaccard index = 0.8603 • Felz: k: 10,20,50,100,200,300,400,500 • SLIC: Regions side: 5,10,20,30,40,50 • SE =7 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8891 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the train set 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8603 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the test set 36
  • 37. Results combining all Felz. and Slic levels Threshold = 0.56 SE = 7 37
  • 38. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 38
  • 39. Type of users and their particularities • Painter: Lot of foreground clicks inside the object to segment 39
  • 40. Type of users and their particularities • Tired: Few clicks per image 40
  • 41. Type of users and their particularities • Border guards: Most of the bg clicks are in the contour of the image. 41
  • 42. Type of users and their particularities • Surrounders: Most of the fg clicks are in the contour of the image 42
  • 43. Type of users and their particularities • Mirrors: Have understood the experiment upside-down 43
  • 44. Type of users and their particularities • Spammers: Randomly placed foreground clicks over the image. 44
  • 45. Type of users and their particularities • Experts: Have well-understood the experiment and just made few mistakes 45
  • 46. Type of users and their particularities • Different pattern: Does not follow the same pattern of clicks in all images 46
  • 47. Manually categorization • It is done a manually categorization by considering just the 5 gold standard images Users Manually categorization 1 Painter 2 Expert 3 Mirror 4 Expert 5 Border guard 6 Expert 7 Tired 8 Border guard 9 Expert 10 Different pattern 11 Different pattern 12 Expert 13 Expert 14 Expert 15 Expert 16 Expert 17 Tired 18 Surrounder 19 Spammer 20 Expert 47
  • 48. Manual rules for automatic user categorization Features Painter The mirror The border guard The surrounder The spammer The tired The expert # clicks >150/image - - - - <5/image - fg clicks(%) >95% - <20% >95% >90% - - errors(%) <3% >90% - - >40% <20% - Jaccard index (%) - <10% - - - <80% >80% Contour fg(%) (fg contour clicks/total fg clicks) - - - >80% <80% - - Contour bg(%) (bg contour clicks/total bg clicks) - - >70% - - - - • According to the particularities of each type of user, a set of features and its rules are created: 48
  • 49. Automatic categorization evaluation for the test set Prediction Painter Mirror Expert Spammer Surrounder Border Guard Tired Diff. Pattern Ground Truth Painter 1 0 0 0 0 0 0 0 Mirror 0 1 0 0 0 0 0 0 Expert 0 0 9 0 0 0 0 1 Spammer 0 0 0 1 0 0 0 0 Surrounder 0 0 0 0 1 0 0 0 Border guard 0 0 0 0 0 1 0 1 Tired 0 0 0 0 0 0 1 1 Diff. pattern 0 0 0 0 0 0 0 2 49
  • 50. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 50
  • 51. Conclusions • Jaccard index is a better measure compared to error rate to separate bad users from good ones 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of users Jaccard index by taking different number of users JaccardIndex Users sorted by its ascendent Jaccard index Users sorted by its descendent error rate 51
  • 52. Conclusions • Better results with partial than with total filtering • Filtering clicks only makes sense when treating with bad users 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard index JaccardIndex Comparing results with partial filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks 0 2 4 6 8 10 12 14 16 18 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of Users sorted by its descended Jaccard index JaccardIndex Comparing results with total filtering and without filtering Felz. sppxl. technique Ncuts spxxl. technique SLIC spxxl. technique With no filtering clicks Partial filtering Total filtering 52
  • 53. Conclusions • In the foreground map algorithm it is reached the best result by combining Felzenzwalb and Slic superpixel techniques with different levels 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8891 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the train set 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 X: 0.56 Y: 0.8603 Threshold JaccardIndex Combining Slic and Felzenzwalb superpixels techniques in the test set 53
  • 54. Conclusions Images from User 11 • It is not possible to automatically categorize users that does not follow the same pattern of clicks in all images 54
  • 55. 1. Motivation 2. Related Work 3. Treatment of human interaction a) Removing human interaction - Combination of object candidates b) Taking advantage of all human interaction - Foreground map algorithm 4. Automatic categorization of the users 5. Conclusions 6. Future work Outline 55
  • 56. Future work • Study different techniques for filtering clicks in a same superpixel. • Take advantage of the clicks of some users to create a better mask (e.g. Border guard and Surrounder users) • Train classifier for automatic user categorization 56