The document proposes a new method called the Clustered Personal Classifier (CPC) method to address quality control issues when learning from crowdsourced data. The CPC method focuses on the similarity between workers by modeling their relationships and fusing similar workers to reduce parameters and provide more robust estimation compared to existing personal classifier methods. Experimental results on both synthetic and real-world data demonstrate that the CPC method outperforms other methods and can detect outlier workers through clustering analysis without the need for validation data.

1. Clustering Crowds
Hiroshi Kajino1, Yuta Tsuboi2, Hisashi Kashima1
1: The University of Tokyo
2: IBM Research - Tokyo
July 16th, 2013 1AAAI-13
*H. Kajino and H. Kashima were supported by the FIRST program.

2. Outline
• Motivation and Problem Setting
Quality control problem of crowdsourcing
• Existing Method
Learning from a crowd-generated training set
• Proposed Method
Focusing on the similarity between workers
• Experimental Results
Robust estimation can be realized
• Conclusion
July 16th, 2013 AAAI-13 2

3. Outline
• Motivation and Problem Setting
Quality control problem of crowdsourcing
• Existing Method
Learning from a crowd-generated training set
• Proposed Method
Focusing on the similarity between workers
• Experimental Results
Robust estimation can be realized
• Conclusion
July 16th, 2013 AAAI-13 3

4. Crowdsourcing
• Crowdsourcing: system to access large crowds
Pros: process human intelligence tasks at low cost
Cons: abilities of workers are unknown
⇒ Quality of results is not guaranteed
July 16th, 2013 AAAI-13 4
Able to access large, but unknown manpower
WorkerRequester
2. Return results
1. Request tasks
3. Pay rewards

5. Task in Machine Learning Community
• Task: picture = bird ?
Pros: easy construct a large training set at low cost
Cons: quality of labels is not guaranteed
July 16th, 2013 AAAI-13 5
Large, but low-quality training set can be obtained easily
Difficult
Easy
Superior Inferior True labels
(Unobservable)
Yes Yes No Yes
No
No
No No
Yes No Yes
No

6. Task in Machine Learning Community
• Task: picture = bird ?
Pros: easy construct a large training set at low cost
Cons: quality of labels is not guaranteed
July 16th, 2013 AAAI-13 6
Large, but low-quality training set can be obtained easily
Difficult
Easy
Superior Inferior True labels
(Unobservable)
Yes Yes No Yes
No
No
No No
Yes No Yes
No
Overcome this difficulty

7. Problem Setting
• Input
– Feature vector : xi ∈RD (i=1,…,I)
– Worker : j ∈{1,2,…,J}
– Crowd label: yij ∈{0,1}
• Output
– classifier w0 ∈ RD (logistic regression model)
Note: we do not use the ground truths
• Common Approach:
1. Model the relation between w0 and {yij}
2. Inferring the model to obtain w0
July 16th, 2013 AAAI-13 7
Estimate a classifier from crowd-generated data
Bird or not
w0

8. Outline
• Motivation and Problem Setting
Quality control problem of crowdsourcing
• Existing Method
Learning from a crowd-generated training set
• Proposed Method
Focusing on the similarity between workers
• Experimental Results
Robust estimation can be realized
• Conclusion
July 16th, 2013 AAAI-13 8

9. • Personal Classifier (PC) Method [Kajino+,12]
– Worker j = classifier wj (= w0 + (noise))
July 16th, 2013 AAAI-13 9
Aggregate “personal classifiers” to obtain w0
personal classifiers
w0 yi2
yi1
true classifier
crowd labels
w1
w2
w3 yi3
N(w0 | 0, η-1I)
j=2
j=3
j=1
prior
distribution
known
unknown
Existing Method

10. • Parameter estimation = MAP estimation
– Parameters: w0, W={wj}J
j=1
– Solve the convex optimization problem:
July 16th, 2013 AAAI-13 10
Parameter estimation = optimizing a convex function
min
w0, W
(logistic loss)
Existing Method
priormodel-relationloss for PCs

11. Existing Method: Discussion
• Personal Classifier Method [Kajino+, 2012]
#(parameters / worker) = D
Pros: global optimum
Cons: bad performance in case of few data per worker
• Clustered Personal Classifier Method
Pros: global optimum & moderate performance
Key: fuse similar workers to decrease the degree of freedom
July 16th, 2013 AAAI-13 11
Estimation can be unstable for the PC method
Proposed

12. Outline
• Motivation and Problem Setting
Quality control problem of crowdsourcing
• Existing Method
Learning from a crowd-generated training set
• Proposed Method
Focusing on the similarity between workers
• Experimental Results
Robust estimation can be realized
• Conclusion
July 16th, 2013 AAAI-13 12

13. Proposed Method: Idea
• Analysis on workers [Welinder+, 2010]
“Notice how the annotators’ decision planes fall roughly
into three clusters”
– Clustering workers is a reasonable idea
(phrase & picture are cited from The multidimensional wisdom of crowds by Welinder+, NIPS 2010)
July 16th, 2013 AAAI-13 13
Similarity between workers can be observed in real data

14. Proposed Method: Formulation
• Clustered Personal Classifier (CPC) Method
– Model-relation term finds and fuses similar workers
→ Cut down the degree of freedom
(μ controls the strength of clustering)
July 16th, 2013 AAAI-13 14
Fuse similar workers to cut down the degree of freedom
(cf. for the PC method)
where forcing wj = wk
model-relation

15. Outline
• Motivation and Problem Setting
Quality control problem of crowdsourcing
• Existing Method
Learning from a crowd-generated training set
• Proposed Method
Focusing on the similarity between workers
• Experimental Results
Robust estimation can be realized
• Conclusion
July 16th, 2013 AAAI-13 15

16. Experiments on Synthetic Data
• Synthetic Data (J=I=10, spammers (random worker) & experts)
(L) (Dimension)=2: PC method = CPC method
(R) (Dimension)=10: PC method < CPC method
July 16th, 2013 AAAI-13 16
Robust performance on a small data set
Percentage of spammers Percentage of spammers
Proposed
Existing
better

17. Experiments on Real Data: Performance
• Performance Test on Real Data [Finin+,10]
– NER task (each word is named entity or not)
– (Dimension)=161,901, #(instances)=17,747, #(workers)=42
July 16th, 2013 AAAI-13 17
Proposed method outperforms other methods
Precision Recall F-measure
CPC method 0.647 0.716 0.680
PC method 0.637 0.721 0.677
LC method 0.625 0.732 0.675
AOC method 0.680 0.670 0.675
MV method 0.686 0.651 0.668
Existing
Method
Proposed

18. Experiments on Real Data: Clustering
• Hierarchical clustering on workers by increasing μ
• Outlier worker can be detected without “honey pots”
July 16th, 2013 AAAI-13 18
Clustering result indicates the existence of an outlier worker
Precision: 0.454
Recall: 0.857
Strength of clustering (=μ) -->

19. Outline
• Motivation and Problem Setting
Quality control problem of crowdsourcing
• Existing Method
Learning from a crowd-generated training set
• Proposed Method
Focusing on the similarity between workers
• Experimental Results
Robust estimation can be realized
• Conclusion
July 16th, 2013 AAAI-13 19

20. Conclusion
• Problem Setting
– Learning from redundant, variable-quality training data
• Problem of the PC Method
– #(parameters) is relatively large
– Unstable when data for one worker are small
• Proposed Method (CPC Method)
– Cut the degree of freedom by fusing similar workers
• Experimental Results
– More robust estimation in case of small data sets
– Valid as a method of “mining” workers
July 16th, 2013 AAAI-13 20
Introducing similarities between workers is beneficial

21. July 16th, 2013 AAAI-13 21