半教師あり学習

17,921 views

Published on

0 Comments
34 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
17,921
On SlideShare
0
From Embeds
0
Number of Embeds
6,531
Actions
Shares
0
Downloads
214
Comments
0
Likes
34
Embeds 0
No embeds

No notes for slide

半教師あり学習

  1. 1. Introduction to Semi- Supervised Learning Thursday, January 21, 2010
  2. 2. Outline • • • Self Traininga • Multiview Algorithms • Generative Models • S3VMs(TSVMs) • Graph-Based Algorithms • • Thursday, January 21, 2010
  3. 3. ? Thursday, January 21, 2010
  4. 4. • • • costly... • • • : Thursday, January 21, 2010
  5. 5. • • : ? => SSL Thursday, January 21, 2010
  6. 6. ? Thursday, January 21, 2010
  7. 7. • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  8. 8. SSL • Self Training • Multiview Algorithms • Generative Models • S3VMs(TSVMs) • Graph-Based Algorithms Thursday, January 21, 2010
  9. 9. Self Training Thursday, January 21, 2010
  10. 10. Self Training • : • : p(y=1|x) 0.99 1 ! • Thursday, January 21, 2010
  11. 11. Self Training • f • f •( ) • Thursday, January 21, 2010
  12. 12. Self Training • • ! • • • • • Thursday, January 21, 2010
  13. 13. Multiview Algorithms Thursday, January 21, 2010
  14. 14. Co-training • Self Training => • ? • Thursday, January 21, 2010
  15. 15. : Web http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  16. 16. • • ) (BOF • • Co-training idea • • Thursday, January 21, 2010
  17. 17. Co-training • • • Thursday, January 21, 2010
  18. 18. Co-training http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  19. 19. Co-training • • • Self Training • • • Thursday, January 21, 2010
  20. 20. Co-training • Co-EM • EM • Fake feature split • • Multiview • • => Thursday, January 21, 2010
  21. 21. Generative Models Thursday, January 21, 2010
  22. 22. Generative Models • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  23. 23. Generative Models • • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  24. 24. Generative Models • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  25. 25. Generative Models • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  26. 26. Generative Models • : • ( ) • Thursday, January 21, 2010
  27. 27. Generative Models • • • EM • • • EM • • • Baum-Welch algorithm Thursday, January 21, 2010
  28. 28. • • • : PRML => EM MLE Thursday, January 21, 2010
  29. 29. • ( ) • y 2 • EM MLE Thursday, January 21, 2010
  30. 30. E-step and M-step http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  31. 31. Generative Models • • • • • • EM ( ) • => page Thursday, January 21, 2010
  32. 32. Generative Models http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  33. 33. • • λ(λ < 1) Thursday, January 21, 2010
  34. 34. S3VMs(TSVMs) Thursday, January 21, 2010
  35. 35. TSVMs • ( ) http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  36. 36. (RKHS) • SVM • => (RKHS:reproducing kernel Hilbert space) • Thursday, January 21, 2010
  37. 37. k k f Thursday, January 21, 2010
  38. 38. • • • ( ) • Thursday, January 21, 2010
  39. 39. • (F ) • x F (F !) Thursday, January 21, 2010
  40. 40. •f F • • f f( x ) => f => F => !! Thursday, January 21, 2010
  41. 41. Thursday, January 21, 2010
  42. 42. SVM RKHS ! hw RKHS Thursday, January 21, 2010
  43. 43. hinge loss hinge loss !! http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  44. 44. • • ? • f(x)>1 or f(x)<-1 • ! => http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  45. 45. • Joachims(1999) • branch- and-bound search( ) • 10000 Thursday, January 21, 2010
  46. 46. S3VM • • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  47. 47. CCCP (Concave-Convex procedure) • Yuille (2003) • • ) ( • Difference of Convex (DC ) • Update Thursday, January 21, 2010
  48. 48. CCCP (Concave-Convex procedure) • update • • (update ) • !! Thursday, January 21, 2010
  49. 49. TSVMs + CCCP • CCCP TSVMs • • Iteration 2 http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf http://www.stat.umn.edu/~xshen/paper/tsvm.pdf Thursday, January 21, 2010
  50. 50. L. Wang (2007) • SVM TSMVs CCCP TSMVs http://www.stat.umn.edu/~xshen/paper/tsvm.pdf Thursday, January 21, 2010
  51. 51. Graph-Based Algorithms Thursday, January 21, 2010
  52. 52. : • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  53. 53. • !! • • etc http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  54. 54. • sparseness http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  55. 55. http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf https://www.aaai.org/Papers/ICML/2003/ICML03-118.pdf Thursday, January 21, 2010
  56. 56. Graph-based SSL • : edge node • : • (instance ) • kNN : k 1 0 => sparse • : • dense Thursday, January 21, 2010
  57. 57. Graph-based SSL • mincut • harmonic • manifold regularization Thursday, January 21, 2010
  58. 58. • • 0 1 fix • Thursday, January 21, 2010
  59. 59. • => • sink source • : http://john.blitzer.com/tutorial/ssl_tutorial.pdf Thursday, January 21, 2010
  60. 60. source sink : source ∞ : sink -∞ Thursday, January 21, 2010
  61. 61. harmonic => Laplacian • 0-1 y • Zhu (2003a) Goldberg (2006) • Thursday, January 21, 2010
  62. 62. harmonic => Laplacian (SVM ) ( L a U b f ... ) → !! Thursday, January 21, 2010
  63. 63. http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf Thursday, January 21, 2010
  64. 64. y_i hat{y}_i http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf Thursday, January 21, 2010
  65. 65. Laplacian •f 0 ! => Thursday, January 21, 2010
  66. 66. Goldberg (2006) • SVMR Metric labeling • SSL • http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf Thursday, January 21, 2010
  67. 67. Manifold regularization • Harmonic • ... • • RKHS • :) Thursday, January 21, 2010
  68. 68. Manifold regularization • • • SVM • ! Thursday, January 21, 2010
  69. 69. : • • 2 • ( )RKHS • Thursday, January 21, 2010
  70. 70. • f • f Thursday, January 21, 2010
  71. 71. • • • Thursday, January 21, 2010
  72. 72. • • Iterated Laplacians • Heat semigroup • • Squared norm of the Hessian • Belkin (2004) http://www.geocities.co.jp/Technopolis/5893/publication/kernel.pdf Thursday, January 21, 2010
  73. 73. Laplacian Regularized Least Squres(LapRLS) • + 0 Regularized Least Squres • closed Thursday, January 21, 2010
  74. 74. Laplacian Support Vector Machines • SVM • 0 SVM Thursday, January 21, 2010
  75. 75. ? • (Harmonic ) • • Thursday, January 21, 2010
  76. 76. Thursday, January 21, 2010
  77. 77. mixture model, EM TSMVs Co-training i.i.d ( Graph-based ) Thursday, January 21, 2010
  78. 78. • no pain, no gain • no model assumption, no gain • wrong model assumption, no gain, a lot of pain Thursday, January 21, 2010
  79. 79. ( | ) • • Co-boosting • bootstrap • Directed graphs • Information Regularization • Structural Learning • Large Data • • PAC Statistical Learning Theory Thursday, January 21, 2010
  80. 80. Thursday, January 21, 2010
  81. 81. (Tutorial etc) • • NAACL 2006 Tutorial: Inductive Semi-supervised Learning with Applicability to NLP, A. Sarkar and G. Haffari. • ICML 2007 Tutorial: Semi-supervised Learning, Xiaojin Zhu. • Blitzer, J. and Zhu, J. (2008). ACL 2008 tutorial on Semi- Supervised learning. http://ssl-acl08.wikidot.com/. • • X. Zhu. Semi-supervised learning literature survey. Technical report, Computer Sciences, University of Wisconsin- Madison, 2007. • Zhu, X. (2005). Semi-supervised learning with graphs. Doctoral dissertation, Carnegie Mellon University. CMU- LTI-05-192. Thursday, January 21, 2010
  82. 82. (Generative model) • Nigam, K., McCallum, A. K., Thrun, S., & Mitchell, T. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39, 103–134. • Liu, B., Lee, W. S.,Yu, P. S., & Li, X. (2002). Partially supervised classification of text documents. Proceedings of the Nineteenth International Conference on Machine Learning (ICML). • Lee, W. S., & Liu, B. (2003). Learning with positive and unlabeled examples using weighted logistic regression. Proceedings of the Twentieth International Conference on Machine Learning (ICML). • Denis, F., Gilleron, R., & Tommasi, M. (2002). Text classification from positive and unlabeled examples. The 9th International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems(IPMU). Thursday, January 21, 2010
  83. 83. (TSVMs) • Joachims, T. (1999). Transductive inference for text classification using support vector machines. Proc. 16th International Conf. on Machine Learning (pp. 200– 209). Morgan Kaufmann, San Francisco, CA. • Yuille, A.L., Rangara jan, A. The concave-convex procedure. Neural Computation 15(4) (2003) 915–936. • L. Wang, X. Shen, and W. Pan. On transductive support vector machines. In J. Verducci, X. Shen, and J. Lafferty, editors, Prediction and Discovery. American Mathematical Society, 2007. • R. Collobert, et al. (2006). Large Scale Transductive SVMs. Journal of Machine Learning Research 7:1687-1712. Thursday, January 21, 2010
  84. 84. (Graph-based) • Blum, A., & Chawla, S. (2001). Learning from labeled and unlabeled data using graph mincuts. Proc. 18th International Conf. on Machine Learning. • Zhu, X., Ghahramani, Z., & Lafferty, J. (2003a). Semi-supervised lear ning using Gaussian fields and harmonic functions. The 20th International Conference on Machine Learning (ICML). • Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22, 888–905. • Pang, B., & Lee, L. (2004). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. Proceedings of the Association for Computational Linguistics (pp. 271–278). • Goldberg, A., & Zhu, X. (2006). Seeing stars when there aren’t many s tars: Graph- based semi-supervised learning for sentiment categorization. HLT-NAACL 2006 Workshop on Textgraphs: Graph-based Algorithms for Natural Language Processing. New York, NY. • Belkin, M., Niyogi, P., & Sindhwani, V. (2004b). Manifold regularization: A geometric framework for learning from examples (Technical Report TR-2004-06). University of Chicago. • M. Belkin & P. Niyogi (2002). `Using Manifold Structure for Partially Labelled Classification'. In NIPS, pp. 929+. Thursday, January 21, 2010
  85. 85. ( ) • Seeger, M. (2001). Learning with labeled and unlabeled data (Technical Report). University of Edinburgh. • François Denis, Bat M, Universit'e De Lille I. PAC Learning from Positive Statistical Queries. Proc. 9th International Conference on Algorithmic Learning Theory - ALT '98 • • Thursday, January 21, 2010

×