Introduction to Semi-
                        Supervised Learning



Thursday, January 21, 2010
Outline
                    •
                    •
                             •   Self Traininga
                      ...
?


Thursday, January 21, 2010
•
                             •
                    •                    costly...
                             •
       ...
•

                    •        :


                                 ? => SSL



Thursday, January 21, 2010
?


Thursday, January 21, 2010
•
                    •




                             http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, Ja...
SSL

                    • Self Training
                    • Multiview Algorithms
                    • Generative Model...
Self Training



Thursday, January 21, 2010
Self Training

                    •                    :


                    •                :             p(y=1|x)   ...
Self Training

                    •
                             f

                    •                f


            ...
Self Training
                    •
                             •         !

                             •
             ...
Multiview Algorithms



Thursday, January 21, 2010
Co-training

                    •             Self Training

                             =>
                    •       ...
: Web




                                     http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, ...
•
                             •      )
                                                  (BOF


                    •
   ...
Co-training

                    •
                    •

                    •


Thursday, January 21, 2010
Co-training




                               http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, ...
Co-training
                    •
                             •
                             •   Self Training
          ...
Co-training
                    • Co-EM
                     •                     EM

                    • Fake feature ...
Generative Models



Thursday, January 21, 2010
Generative Models

                    •


                    •


                                      http://pages.cs.w...
Generative Models
                    •


                    •

                    •

                                  ...
Generative Models


                    •




                                      http://pages.cs.wisc.edu/~jerryzhu/pub...
Generative Models

                    •

                    •



                                      http://pages.cs.w...
Generative Models


                    •            :


                    •                (   )


                    ...
Generative Models
                    •
                             •
                             •   EM

              ...
•
                    •
                    •        : PRML
                                      => EM   MLE




Thursday...
•                           (
                                   )

                    •                  y            2
...
E-step and M-step




                                      http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday,...
Generative Models
                    •
                             •
                             •
                    ...
Generative Models




                                      http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf



Thursd...
•

                    •
                             λ(λ < 1)




Thursday, January 21, 2010
S3VMs(TSVMs)



Thursday, January 21, 2010
TSVMs
                    •
                             (                    )




                                 http:...
(RKHS)

                    • SVM
                    •
                                 =>
                             (...
k



                                     k


                             f




Thursday, January 21, 2010
•
                             •
                    •                (
                                 )
               ...
•
                             (F        )

                    •                          x
                             ...
•f         F



                    •
                    •              f               f(   x
                          ...
Thursday, January 21, 2010
SVM


                                   RKHS   !




             hw       RKHS




Thursday, January 21, 2010
hinge loss




                                               hinge loss                          !!




                 ...
•
                 •                         ?

                       •     f(x)>1 or f(x)<-1


                       •
...
• Joachims(1999)
                    •                                branch-
                             and-bound searc...
S3VM

                    •

                             •
                    •
                                   http:...
CCCP
                        (Concave-Convex procedure)
                    •        Yuille (2003)

                    •
...
CCCP
                        (Concave-Convex procedure)
                    • update
                     •
              ...
TSVMs + CCCP
                    • CCCP           TSVMs

                    •
                    • Iteration      2




...
L. Wang (2007)
                    •                                           SVM
                                       ...
Graph-Based
                              Algorithms


Thursday, January 21, 2010
:
                    •
                    •




                                 http://pages.cs.wisc.edu/~jerryzhu/pub/...
•                     !!
                             •
                             •   etc




                         ...
•        sparseness




                             http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, Januar...
http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf

                             https://www.aaai.org/Papers/ICML/2003/I...
Graph-based SSL
                    •                :                   edge           node


                    •      ...
Graph-based SSL



                    • mincut
                    • harmonic
                    • manifold regularizati...
•
                    •        0   1
                                     fix



                    •

Thursday, January 2...
•                                           =>
                             •   sink
                                     ...
source                          sink




              : source       ∞   : sink      -∞



Thursday, January 21, 2010
harmonic => Laplacian
                    •                      0-1           y

                    • Zhu        (2003a)...
harmonic => Laplacian
                                         (SVM   )




                 (
                           ...
http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf
Thursday, January 21, 2010
y_i
         hat{y}_i




                             http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs....
Laplacian


                    •f
                                    0
                             ! =>




Thursday, J...
Goldberg (2006)

                    •        SVMR Metric labeling


                    •                         SSL


 ...
Manifold regularization
                    • Harmonic
                     •                     ...
                    ...
Manifold regularization
                    •
                             •
                    • SVM

                  ...
:
                    •


                             •       2
                    •            (       )RKHS

         ...
•
                             f



                    •            f




Thursday, January 21, 2010
•


                    •

                    •


Thursday, January 21, 2010
•

                             • Iterated Laplacians
                             • Heat semigroup
                      ...
Laplacian Regularized
                       Least Squres(LapRLS)
                    •
                             +
   ...
Laplacian Support
                             Vector Machines
                    • SVM

                    •

         ...
?
                    •            (Harmonic
                             )
                    •
                    •


...
Thursday, January 21, 2010
mixture model, EM

                                  TSMVs

                                Co-training
                  ...
• no pain, no gain
                    • no model assumption, no gain
                    • wrong model assumption, no gai...
(      | )
                    •
                             •   Co-boosting

                             •   bootstrap
...
Thursday, January 21, 2010
(Tutorial etc)
                    •
                             •   NAACL 2006 Tutorial: Inductive Semi-supervised Learn...
(Generative model)
                    •        Nigam, K., McCallum, A. K., Thrun, S., & Mitchell, T. (2000). Text
       ...
(TSVMs)
                    •        Joachims, T. (1999). Transductive inference for text classification
                  ...
(Graph-based)
                    •        Blum, A., & Chawla, S. (2001). Learning from labeled and unlabeled data using g...
(                    )
                    •        Seeger, M. (2001). Learning with labeled and unlabeled
               ...
Upcoming SlideShare
Loading in...5
×

半教師あり学習

7,945

Published on

0 Comments
22 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,945
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
161
Comments
0
Likes
22
Embeds 0
No embeds

No notes for slide

Transcript of "半教師あり学習"

  1. 1. Introduction to Semi- Supervised Learning Thursday, January 21, 2010
  2. 2. Outline • • • Self Traininga • Multiview Algorithms • Generative Models • S3VMs(TSVMs) • Graph-Based Algorithms • • Thursday, January 21, 2010
  3. 3. ? Thursday, January 21, 2010
  4. 4. • • • costly... • • • : Thursday, January 21, 2010
  5. 5. • • : ? => SSL Thursday, January 21, 2010
  6. 6. ? Thursday, January 21, 2010
  7. 7. • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  8. 8. SSL • Self Training • Multiview Algorithms • Generative Models • S3VMs(TSVMs) • Graph-Based Algorithms Thursday, January 21, 2010
  9. 9. Self Training Thursday, January 21, 2010
  10. 10. Self Training • : • : p(y=1|x) 0.99 1 ! • Thursday, January 21, 2010
  11. 11. Self Training • f • f •( ) • Thursday, January 21, 2010
  12. 12. Self Training • • ! • • • • • Thursday, January 21, 2010
  13. 13. Multiview Algorithms Thursday, January 21, 2010
  14. 14. Co-training • Self Training => • ? • Thursday, January 21, 2010
  15. 15. : Web http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  16. 16. • • ) (BOF • • Co-training idea • • Thursday, January 21, 2010
  17. 17. Co-training • • • Thursday, January 21, 2010
  18. 18. Co-training http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  19. 19. Co-training • • • Self Training • • • Thursday, January 21, 2010
  20. 20. Co-training • Co-EM • EM • Fake feature split • • Multiview • • => Thursday, January 21, 2010
  21. 21. Generative Models Thursday, January 21, 2010
  22. 22. Generative Models • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  23. 23. Generative Models • • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  24. 24. Generative Models • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  25. 25. Generative Models • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  26. 26. Generative Models • : • ( ) • Thursday, January 21, 2010
  27. 27. Generative Models • • • EM • • • EM • • • Baum-Welch algorithm Thursday, January 21, 2010
  28. 28. • • • : PRML => EM MLE Thursday, January 21, 2010
  29. 29. • ( ) • y 2 • EM MLE Thursday, January 21, 2010
  30. 30. E-step and M-step http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  31. 31. Generative Models • • • • • • EM ( ) • => page Thursday, January 21, 2010
  32. 32. Generative Models http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  33. 33. • • λ(λ < 1) Thursday, January 21, 2010
  34. 34. S3VMs(TSVMs) Thursday, January 21, 2010
  35. 35. TSVMs • ( ) http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  36. 36. (RKHS) • SVM • => (RKHS:reproducing kernel Hilbert space) • Thursday, January 21, 2010
  37. 37. k k f Thursday, January 21, 2010
  38. 38. • • • ( ) • Thursday, January 21, 2010
  39. 39. • (F ) • x F (F !) Thursday, January 21, 2010
  40. 40. •f F • • f f( x ) => f => F => !! Thursday, January 21, 2010
  41. 41. Thursday, January 21, 2010
  42. 42. SVM RKHS ! hw RKHS Thursday, January 21, 2010
  43. 43. hinge loss hinge loss !! http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  44. 44. • • ? • f(x)>1 or f(x)<-1 • ! => http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  45. 45. • Joachims(1999) • branch- and-bound search( ) • 10000 Thursday, January 21, 2010
  46. 46. S3VM • • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  47. 47. CCCP (Concave-Convex procedure) • Yuille (2003) • • ) ( • Difference of Convex (DC ) • Update Thursday, January 21, 2010
  48. 48. CCCP (Concave-Convex procedure) • update • • (update ) • !! Thursday, January 21, 2010
  49. 49. TSVMs + CCCP • CCCP TSVMs • • Iteration 2 http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf http://www.stat.umn.edu/~xshen/paper/tsvm.pdf Thursday, January 21, 2010
  50. 50. L. Wang (2007) • SVM TSMVs CCCP TSMVs http://www.stat.umn.edu/~xshen/paper/tsvm.pdf Thursday, January 21, 2010
  51. 51. Graph-Based Algorithms Thursday, January 21, 2010
  52. 52. : • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  53. 53. • !! • • etc http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  54. 54. • sparseness http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  55. 55. http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf https://www.aaai.org/Papers/ICML/2003/ICML03-118.pdf Thursday, January 21, 2010
  56. 56. Graph-based SSL • : edge node • : • (instance ) • kNN : k 1 0 => sparse • : • dense Thursday, January 21, 2010
  57. 57. Graph-based SSL • mincut • harmonic • manifold regularization Thursday, January 21, 2010
  58. 58. • • 0 1 fix • Thursday, January 21, 2010
  59. 59. • => • sink source • : http://john.blitzer.com/tutorial/ssl_tutorial.pdf Thursday, January 21, 2010
  60. 60. source sink : source ∞ : sink -∞ Thursday, January 21, 2010
  61. 61. harmonic => Laplacian • 0-1 y • Zhu (2003a) Goldberg (2006) • Thursday, January 21, 2010
  62. 62. harmonic => Laplacian (SVM ) ( L a U b f ... ) → !! Thursday, January 21, 2010
  63. 63. http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf Thursday, January 21, 2010
  64. 64. y_i hat{y}_i http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf Thursday, January 21, 2010
  65. 65. Laplacian •f 0 ! => Thursday, January 21, 2010
  66. 66. Goldberg (2006) • SVMR Metric labeling • SSL • http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf Thursday, January 21, 2010
  67. 67. Manifold regularization • Harmonic • ... • • RKHS • :) Thursday, January 21, 2010
  68. 68. Manifold regularization • • • SVM • ! Thursday, January 21, 2010
  69. 69. : • • 2 • ( )RKHS • Thursday, January 21, 2010
  70. 70. • f • f Thursday, January 21, 2010
  71. 71. • • • Thursday, January 21, 2010
  72. 72. • • Iterated Laplacians • Heat semigroup • • Squared norm of the Hessian • Belkin (2004) http://www.geocities.co.jp/Technopolis/5893/publication/kernel.pdf Thursday, January 21, 2010
  73. 73. Laplacian Regularized Least Squres(LapRLS) • + 0 Regularized Least Squres • closed Thursday, January 21, 2010
  74. 74. Laplacian Support Vector Machines • SVM • 0 SVM Thursday, January 21, 2010
  75. 75. ? • (Harmonic ) • • Thursday, January 21, 2010
  76. 76. Thursday, January 21, 2010
  77. 77. mixture model, EM TSMVs Co-training i.i.d ( Graph-based ) Thursday, January 21, 2010
  78. 78. • no pain, no gain • no model assumption, no gain • wrong model assumption, no gain, a lot of pain Thursday, January 21, 2010
  79. 79. ( | ) • • Co-boosting • bootstrap • Directed graphs • Information Regularization • Structural Learning • Large Data • • PAC Statistical Learning Theory Thursday, January 21, 2010
  80. 80. Thursday, January 21, 2010
  81. 81. (Tutorial etc) • • NAACL 2006 Tutorial: Inductive Semi-supervised Learning with Applicability to NLP, A. Sarkar and G. Haffari. • ICML 2007 Tutorial: Semi-supervised Learning, Xiaojin Zhu. • Blitzer, J. and Zhu, J. (2008). ACL 2008 tutorial on Semi- Supervised learning. http://ssl-acl08.wikidot.com/. • • X. Zhu. Semi-supervised learning literature survey. Technical report, Computer Sciences, University of Wisconsin- Madison, 2007. • Zhu, X. (2005). Semi-supervised learning with graphs. Doctoral dissertation, Carnegie Mellon University. CMU- LTI-05-192. Thursday, January 21, 2010
  82. 82. (Generative model) • Nigam, K., McCallum, A. K., Thrun, S., & Mitchell, T. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39, 103–134. • Liu, B., Lee, W. S.,Yu, P. S., & Li, X. (2002). Partially supervised classification of text documents. Proceedings of the Nineteenth International Conference on Machine Learning (ICML). • Lee, W. S., & Liu, B. (2003). Learning with positive and unlabeled examples using weighted logistic regression. Proceedings of the Twentieth International Conference on Machine Learning (ICML). • Denis, F., Gilleron, R., & Tommasi, M. (2002). Text classification from positive and unlabeled examples. The 9th International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems(IPMU). Thursday, January 21, 2010
  83. 83. (TSVMs) • Joachims, T. (1999). Transductive inference for text classification using support vector machines. Proc. 16th International Conf. on Machine Learning (pp. 200– 209). Morgan Kaufmann, San Francisco, CA. • Yuille, A.L., Rangara jan, A. The concave-convex procedure. Neural Computation 15(4) (2003) 915–936. • L. Wang, X. Shen, and W. Pan. On transductive support vector machines. In J. Verducci, X. Shen, and J. Lafferty, editors, Prediction and Discovery. American Mathematical Society, 2007. • R. Collobert, et al. (2006). Large Scale Transductive SVMs. Journal of Machine Learning Research 7:1687-1712. Thursday, January 21, 2010
  84. 84. (Graph-based) • Blum, A., & Chawla, S. (2001). Learning from labeled and unlabeled data using graph mincuts. Proc. 18th International Conf. on Machine Learning. • Zhu, X., Ghahramani, Z., & Lafferty, J. (2003a). Semi-supervised lear ning using Gaussian fields and harmonic functions. The 20th International Conference on Machine Learning (ICML). • Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22, 888–905. • Pang, B., & Lee, L. (2004). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. Proceedings of the Association for Computational Linguistics (pp. 271–278). • Goldberg, A., & Zhu, X. (2006). Seeing stars when there aren’t many s tars: Graph- based semi-supervised learning for sentiment categorization. HLT-NAACL 2006 Workshop on Textgraphs: Graph-based Algorithms for Natural Language Processing. New York, NY. • Belkin, M., Niyogi, P., & Sindhwani, V. (2004b). Manifold regularization: A geometric framework for learning from examples (Technical Report TR-2004-06). University of Chicago. • M. Belkin & P. Niyogi (2002). `Using Manifold Structure for Partially Labelled Classification'. In NIPS, pp. 929+. Thursday, January 21, 2010
  85. 85. ( ) • Seeger, M. (2001). Learning with labeled and unlabeled data (Technical Report). University of Edinburgh. • François Denis, Bat M, Universit'e De Lille I. PAC Learning from Positive Statistical Queries. Proc. 9th International Conference on Algorithmic Learning Theory - ALT '98 • • Thursday, January 21, 2010
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×