Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Combining Committee-ba...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Overview
2 / 24
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Semi-Supervised Learni...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Semi-Supervised Learni...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
How can unlabeled data...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
How can unlabeled data...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Self-Training
But the ...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Multi-View Co-Training...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Multi-View Co-Training...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Single-View Co-Trainin...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Single-View Co-Trainin...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
How to measure confiden...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Improving CPE of Decis...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Estimating local compe...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Estimating local compe...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Handwritten Digits Rec...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Experimental Setup
WEK...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Experimental Results
C...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Experimental Results
•...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Combining QBC and CoBC...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Experimental Results
•...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Conclusion
A new singl...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Future Work
Influence o...
Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work
Thanks for your attent...
Upcoming SlideShare
Loading in …5
×

Combining Committee-Based Semi-supervised and Active Learning and Its Application to Handwritten Digits Recognition

374 views
261 views

Published on

Semi-supervised learning reduces the cost of labeling the
training data of a supervised learning algorithm through using unlabeled
data together with labeled data to improve the performance. Co-Training
is a popular semi-supervised learning algorithm, that requires multiple redundant
and independent sets of features (views). In many real-world application
domains, this requirement can not be satisfied. In this paper, a
single-view variant of Co-Training, CoBC (Co-Training by Committee),
is proposed, which requires an ensemble of diverse classifiers instead of
the redundant and independent views. Then we introduce two new learning
algorithms, QBC-then-CoBC and QBC-with-CoBC, which combines
the merits of committee-based semi-supervised learning and committeebased
active learning. An empirical study on handwritten digit recognition
is conducted where the random subspace method (RSM) is used to
create ensembles of diverse C4.5 decision trees. Experiments show that
these two combinations outperform the other non committee-based ones.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
374
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Combining Committee-Based Semi-supervised and Active Learning and Its Application to Handwritten Digits Recognition

  1. 1. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Combining Committee-based Semi-supervised and Active Learning and Its Application to Handwritten Digits Recognition Mohamed Farouk Abdel Hady, Friedhelm Schwenker Institute of Neural Information Processing University of Ulm, Germany {mohamed.abdel-hady|friedhelm.schwenker}@uni-ulm.de April 8, 2010 1 / 24
  2. 2. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Overview 2 / 24
  3. 3. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Semi-Supervised Learning In many domains, the amount of training examples is large but unlabeled. Data labeling process is often tedious, expensive and time consuming because it requires the effort of human experts. Research directions of SSL Semi-Supervised Clustering Semi-Supervised Classification Semi-Supervised Regression Semi-Supervised Dimensionality Reduction 3 / 24
  4. 4. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Semi-Supervised Learning Description SSL algorithm Single-view, Single-learner EM (Nigam and Ghani, 2000) Single-classifier Self-Training (Nigam and Ghani, 2000) Multi-view, Single-learner Co-EM (Nigam and Ghani, 2000) Multiple classifiers Co-Training (Blum and Mitchell, COLT’98) Single-view, Multi-learner Statistical Co-Learning (Goldman et al., 2000) Multiple classifiers Democratic Co-Learning (Y. Zhou et al., 2004) Single-view, Single-learner Tri-Training (Z.-H. Zhou, TKDE’05) Multiple classifiers Co-Forest (Li and Z.-H. Zhou, TSMC’07) Co-Training by Committee Z.-H. Zhou and M. Li, Semi-supervised learning by disagreement, Knowledge and Information Systems, in press. 4 / 24
  5. 5. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work How can unlabeled data be helpful? 5 / 24
  6. 6. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work How can unlabeled data be helpful? 6 / 24
  7. 7. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Self-Training But the most confident examples often lie away from the target decision boundary (non informative examples). Therefore, in many cases this process does not create representative training sets as it selects non informative examples. 7 / 24
  8. 8. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Multi-View Co-Training Blum and Mitchell (1998) As any multi-view learning algorithm, it requires that each training example is represented by multiple sufficient and redundant views, i.e. two or more sets of features that are conditionally independent given the class label and each is sufficient for learning. For web page classification: 1) the text appearing on the page itself, and 2) the text attached to hyperlinks pointing to this page, from other pages. 8 / 24
  9. 9. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Multi-View Co-Training 9 / 24
  10. 10. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Single-View Co-Training by Committee Contribution A single-view variant of Co-Training for application domains in which there are not redundant and independent views is proposed. Two learning frameworks for combining the merits of active learning with semi-supervised learning. Motivation For many real-world applications, the requirement for two sufficient and independent views can not be fulfilled. Co-Training does not work well without an appropriate feature splitting (Nigam and Ghani, 2000) Measuring the labeling confidence is not a straightforward task. 10 / 24
  11. 11. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Single-View Co-Training By Committee 11 / 24
  12. 12. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work How to measure confidence Inaccurate confidence estimation → selecting and adding mislabeled examples to the training set → degrade the classification accuracy Estimating Class Probabilities (CPE) provided by companion committee. Confidence(xu, H (t−1) i ) = max 1≤c≤C H (t−1) i (xu, ωc) Unfortunately, in many cases the classifier does not provide an accurate CPE. For instance, a decision tree provides piecewise constant probability estimates. That is, all unlabeled examples xu which lie into a particular leaf, will have the same CPEs because the exact value of xu is not used in determining its CPE. 12 / 24
  13. 13. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Improving CPE of Decision Trees Laplace Correction, Probability Estimation Tree (PET), (Provost, Machine Learning 2003) P(ωc|xu) = nc + 1 N + C Bagging of PET Retrofitting Decision Tree Classifiers Using Kernel Density Estimation (Fayyad, ICML’95) Improve Decision Trees for Probability-Based Ranking by Lazy Learners (Liang, ICTAI’06) 13 / 24
  14. 14. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Estimating local competence The local competency of an unlabeled example xu given H (t−1) i is defined as follows: Comp(xu, H (t−1) i ) = xn∈N(xu),xn∈ωpred H (t−1) i (xn, ωpred ) ||xn − xu||2 + where ωpred is the class label assigned to xu by H (t−1) i ; H (t−1) j (xn, ωpred ) is the probability given by H (t−1) j that neighbor xn belongs to class ωpred ; is a constant added to avoid zero denominator. It is inspired by decision-dependent distance-based k-nn estimate of the competence that was proposed for dynamic classifier selection. (Woods, PAMI’97) 14 / 24
  15. 15. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Estimating local competence estimating local competence of an unlabeled example given companion committee 15 / 24
  16. 16. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Handwritten Digits Recognition The Handwritten Digits that are described by four sets of features and are publicly available at UCI Repository. The digits were extracted from a collection of Dutch utility maps. A total of 2,000 patterns (200 patterns per class) have been digitized in binary images. Name Description mfeat-pix 240 pixel averages in 2 x 3 windows mfeat-kar 64 Karhunen-Love coefficients mfeat-fac 216 profile correlations mfeat-fou 76 Fourier coefficients of the character shapes 16 / 24
  17. 17. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Experimental Setup WEKA 4 runs of 10-fold cross-validation For SSL, 10% of the training examples (180 patterns) are randomly selected as the initial labeled data set L while the remaining are used as unlabeled data set U. The Random Subspace Method constructs an ensemble of ten C4.5 pruned decision trees (with Laplace Correction) where each tree uses only 50% of the features. We set the pool size u = 100, the sample size n = one and the number of nearest neighbors used to estimate local competence k is 10. 17 / 24
  18. 18. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Experimental Results Comparison between forests and individual trees. Comparison between CoBC and Self-Training. Comparison between CPE and local competence confidence measures. Comparison between CoBC and Co-Forest. 18 / 24
  19. 19. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Experimental Results • : corrected paired t-test implemented in WEKA at 0.05 significance level. 19 / 24
  20. 20. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Combining QBC and CoBC Both semi-supervised learning and active learning tackle the same problem but from different directions. QBC-then-CoBC: QBC provides CoBC with a better starting point instead of randomly selecting labeled examples. QBC-with-CoBC: In QBC-then-CoBC, QBC does not benefit from CoBC. On the other hand, in QBC-with-CoBC, both algorithms are benefiting from each other. 20 / 24
  21. 21. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Experimental Results • : corrected paired t-test implemented in WEKA at 0.05 significance level. 21 / 24
  22. 22. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Conclusion A new single-view committe-based semi-supervised learning framework is proposed. An ensemble of diverse and accurate classifiers can effectively exploit the unlabeled data to improve the recognition accuracy. The random subspace method not only enforces the diversity but also reduces the dimensionality which is desirable in case of small training set size. CoBC outperforms Self-Training. The local competence estimates is an effective confidence measure that outperforms the class probability estimates for sample selection. 22 / 24
  23. 23. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Future Work Influence of ensemble size, random subspace size Different ensemble learners, base learners such as SVM or kNN CoBC depends only on the companion committee H (t−1) j constructed at the previous iteration to measure confidence. We will study the influence of depending on all the previous versions (H (t ) j , t = t − 1, t − 2, . . . , 0). 23 / 24
  24. 24. Overview Semi-Supervised Learning(SSL) Single-View CoBC Experimental Results Conclusion Future Work Thanks for your attention Questions ?? 24 / 24

×