Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

半教師あり学習

25,262 views

Published on

  • Be the first to comment

半教師あり学習

  1. 1. Introduction to Semi- Supervised Learning Thursday, January 21, 2010
  2. 2. Outline • • • Self Traininga • Multiview Algorithms • Generative Models • S3VMs(TSVMs) • Graph-Based Algorithms • • Thursday, January 21, 2010
  3. 3. ? Thursday, January 21, 2010
  4. 4. • • • costly... • • • : Thursday, January 21, 2010
  5. 5. • • : ? => SSL Thursday, January 21, 2010
  6. 6. ? Thursday, January 21, 2010
  7. 7. • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  8. 8. SSL • Self Training • Multiview Algorithms • Generative Models • S3VMs(TSVMs) • Graph-Based Algorithms Thursday, January 21, 2010
  9. 9. Self Training Thursday, January 21, 2010
  10. 10. Self Training • : • : p(y=1|x) 0.99 1 ! • Thursday, January 21, 2010
  11. 11. Self Training • f • f •( ) • Thursday, January 21, 2010
  12. 12. Self Training • • ! • • • • • Thursday, January 21, 2010
  13. 13. Multiview Algorithms Thursday, January 21, 2010
  14. 14. Co-training • Self Training => • ? • Thursday, January 21, 2010
  15. 15. : Web http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  16. 16. • • ) (BOF • • Co-training idea • • Thursday, January 21, 2010
  17. 17. Co-training • • • Thursday, January 21, 2010
  18. 18. Co-training http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  19. 19. Co-training • • • Self Training • • • Thursday, January 21, 2010
  20. 20. Co-training • Co-EM • EM • Fake feature split • • Multiview • • => Thursday, January 21, 2010
  21. 21. Generative Models Thursday, January 21, 2010
  22. 22. Generative Models • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  23. 23. Generative Models • • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  24. 24. Generative Models • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  25. 25. Generative Models • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  26. 26. Generative Models • : • ( ) • Thursday, January 21, 2010
  27. 27. Generative Models • • • EM • • • EM • • • Baum-Welch algorithm Thursday, January 21, 2010
  28. 28. • • • : PRML => EM MLE Thursday, January 21, 2010
  29. 29. • ( ) • y 2 • EM MLE Thursday, January 21, 2010
  30. 30. E-step and M-step http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  31. 31. Generative Models • • • • • • EM ( ) • => page Thursday, January 21, 2010
  32. 32. Generative Models http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  33. 33. • • λ(λ < 1) Thursday, January 21, 2010
  34. 34. S3VMs(TSVMs) Thursday, January 21, 2010
  35. 35. TSVMs • ( ) http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  36. 36. (RKHS) • SVM • => (RKHS:reproducing kernel Hilbert space) • Thursday, January 21, 2010
  37. 37. k k f Thursday, January 21, 2010
  38. 38. • • • ( ) • Thursday, January 21, 2010
  39. 39. • (F ) • x F (F !) Thursday, January 21, 2010
  40. 40. •f F • • f f( x ) => f => F => !! Thursday, January 21, 2010
  41. 41. Thursday, January 21, 2010
  42. 42. SVM RKHS ! hw RKHS Thursday, January 21, 2010
  43. 43. hinge loss hinge loss !! http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  44. 44. • • ? • f(x)>1 or f(x)<-1 • ! => http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  45. 45. • Joachims(1999) • branch- and-bound search( ) • 10000 Thursday, January 21, 2010
  46. 46. S3VM • • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  47. 47. CCCP (Concave-Convex procedure) • Yuille (2003) • • ) ( • Difference of Convex (DC ) • Update Thursday, January 21, 2010
  48. 48. CCCP (Concave-Convex procedure) • update • • (update ) • !! Thursday, January 21, 2010
  49. 49. TSVMs + CCCP • CCCP TSVMs • • Iteration 2 http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf http://www.stat.umn.edu/~xshen/paper/tsvm.pdf Thursday, January 21, 2010
  50. 50. L. Wang (2007) • SVM TSMVs CCCP TSMVs http://www.stat.umn.edu/~xshen/paper/tsvm.pdf Thursday, January 21, 2010
  51. 51. Graph-Based Algorithms Thursday, January 21, 2010
  52. 52. : • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  53. 53. • !! • • etc http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  54. 54. • sparseness http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  55. 55. http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf https://www.aaai.org/Papers/ICML/2003/ICML03-118.pdf Thursday, January 21, 2010
  56. 56. Graph-based SSL • : edge node • : • (instance ) • kNN : k 1 0 => sparse • : • dense Thursday, January 21, 2010
  57. 57. Graph-based SSL • mincut • harmonic • manifold regularization Thursday, January 21, 2010
  58. 58. • • 0 1 fix • Thursday, January 21, 2010
  59. 59. • => • sink source • : http://john.blitzer.com/tutorial/ssl_tutorial.pdf Thursday, January 21, 2010
  60. 60. source sink : source ∞ : sink -∞ Thursday, January 21, 2010
  61. 61. harmonic => Laplacian • 0-1 y • Zhu (2003a) Goldberg (2006) • Thursday, January 21, 2010
  62. 62. harmonic => Laplacian (SVM ) ( L a U b f ... ) → !! Thursday, January 21, 2010
  63. 63. http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf Thursday, January 21, 2010
  64. 64. y_i hat{y}_i http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf Thursday, January 21, 2010
  65. 65. Laplacian •f 0 ! => Thursday, January 21, 2010
  66. 66. Goldberg (2006) • SVMR Metric labeling • SSL • http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf Thursday, January 21, 2010
  67. 67. Manifold regularization • Harmonic • ... • • RKHS • :) Thursday, January 21, 2010
  68. 68. Manifold regularization • • • SVM • ! Thursday, January 21, 2010
  69. 69. : • • 2 • ( )RKHS • Thursday, January 21, 2010
  70. 70. • f • f Thursday, January 21, 2010
  71. 71. • • • Thursday, January 21, 2010
  72. 72. • • Iterated Laplacians • Heat semigroup • • Squared norm of the Hessian • Belkin (2004) http://www.geocities.co.jp/Technopolis/5893/publication/kernel.pdf Thursday, January 21, 2010
  73. 73. Laplacian Regularized Least Squres(LapRLS) • + 0 Regularized Least Squres • closed Thursday, January 21, 2010
  74. 74. Laplacian Support Vector Machines • SVM • 0 SVM Thursday, January 21, 2010
  75. 75. ? • (Harmonic ) • • Thursday, January 21, 2010
  76. 76. Thursday, January 21, 2010
  77. 77. mixture model, EM TSMVs Co-training i.i.d ( Graph-based ) Thursday, January 21, 2010
  78. 78. • no pain, no gain • no model assumption, no gain • wrong model assumption, no gain, a lot of pain Thursday, January 21, 2010
  79. 79. ( | ) • • Co-boosting • bootstrap • Directed graphs • Information Regularization • Structural Learning • Large Data • • PAC Statistical Learning Theory Thursday, January 21, 2010
  80. 80. Thursday, January 21, 2010
  81. 81. (Tutorial etc) • • NAACL 2006 Tutorial: Inductive Semi-supervised Learning with Applicability to NLP, A. Sarkar and G. Haffari. • ICML 2007 Tutorial: Semi-supervised Learning, Xiaojin Zhu. • Blitzer, J. and Zhu, J. (2008). ACL 2008 tutorial on Semi- Supervised learning. http://ssl-acl08.wikidot.com/. • • X. Zhu. Semi-supervised learning literature survey. Technical report, Computer Sciences, University of Wisconsin- Madison, 2007. • Zhu, X. (2005). Semi-supervised learning with graphs. Doctoral dissertation, Carnegie Mellon University. CMU- LTI-05-192. Thursday, January 21, 2010
  82. 82. (Generative model) • Nigam, K., McCallum, A. K., Thrun, S., & Mitchell, T. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39, 103–134. • Liu, B., Lee, W. S.,Yu, P. S., & Li, X. (2002). Partially supervised classification of text documents. Proceedings of the Nineteenth International Conference on Machine Learning (ICML). • Lee, W. S., & Liu, B. (2003). Learning with positive and unlabeled examples using weighted logistic regression. Proceedings of the Twentieth International Conference on Machine Learning (ICML). • Denis, F., Gilleron, R., & Tommasi, M. (2002). Text classification from positive and unlabeled examples. The 9th International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems(IPMU). Thursday, January 21, 2010
  83. 83. (TSVMs) • Joachims, T. (1999). Transductive inference for text classification using support vector machines. Proc. 16th International Conf. on Machine Learning (pp. 200– 209). Morgan Kaufmann, San Francisco, CA. • Yuille, A.L., Rangara jan, A. The concave-convex procedure. Neural Computation 15(4) (2003) 915–936. • L. Wang, X. Shen, and W. Pan. On transductive support vector machines. In J. Verducci, X. Shen, and J. Lafferty, editors, Prediction and Discovery. American Mathematical Society, 2007. • R. Collobert, et al. (2006). Large Scale Transductive SVMs. Journal of Machine Learning Research 7:1687-1712. Thursday, January 21, 2010
  84. 84. (Graph-based) • Blum, A., & Chawla, S. (2001). Learning from labeled and unlabeled data using graph mincuts. Proc. 18th International Conf. on Machine Learning. • Zhu, X., Ghahramani, Z., & Lafferty, J. (2003a). Semi-supervised lear ning using Gaussian fields and harmonic functions. The 20th International Conference on Machine Learning (ICML). • Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22, 888–905. • Pang, B., & Lee, L. (2004). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. Proceedings of the Association for Computational Linguistics (pp. 271–278). • Goldberg, A., & Zhu, X. (2006). Seeing stars when there aren’t many s tars: Graph- based semi-supervised learning for sentiment categorization. HLT-NAACL 2006 Workshop on Textgraphs: Graph-based Algorithms for Natural Language Processing. New York, NY. • Belkin, M., Niyogi, P., & Sindhwani, V. (2004b). Manifold regularization: A geometric framework for learning from examples (Technical Report TR-2004-06). University of Chicago. • M. Belkin & P. Niyogi (2002). `Using Manifold Structure for Partially Labelled Classification'. In NIPS, pp. 929+. Thursday, January 21, 2010
  85. 85. ( ) • Seeger, M. (2001). Learning with labeled and unlabeled data (Technical Report). University of Edinburgh. • François Denis, Bat M, Universit'e De Lille I. PAC Learning from Positive Statistical Queries. Proc. 9th International Conference on Algorithmic Learning Theory - ALT '98 • • Thursday, January 21, 2010

×