This document discusses topic models, LDA, and related concepts. It begins with an overview of LDA and how it uses a graphical model approach for unsupervised learning. Various inference methods for LDA are discussed, including variational inference and Gibbs sampling. The document also covers extensions like correlated topic models and dynamic topic models, as well as applications and researchers in the field. Key concepts covered include posterior approximation, sampling, variational methods, and optimization.
2. LDA Linear Discriminant Analysis Fisher Linear Discriminant Analysis 有监督学习 寻找使类间与类内比最大的投影方向 从矩阵的角度 Latent DirichletAllocation 无监督学习 从图模型的角度 Just make sure we are on the same page
21. David Barber Bayesian Reasoning and Machine Learning Daphne Koller,NirFriedman Probabilistic Graphical Model Bishop Pattern Recognition and Machine Learning Ch8 Eric Xing Probabilistic Graphical Models 获得更多 Bayesian Network : where to learn more
33. Goal The goal is to find short descriptions of the members of a collection that enable efficient processing of large collections while preserving the essential statistical relationships that are useful for basic tasks such as classification, novelty detection, summarization, and similarity and relevance judgments. Goal and Motivation of Topic Model
51. LDA : Topic Five topics from a 50-topic LDA model fit to Science from 1980– 2002 LDA : Five topics from a 50-topic LDA model fit to Science from 1980– 2002
52. LDA : Personas Demo http://personas.media.mit.edu/personasWeb.html LDA : Personas
53. LDA :获得更多 David M Blei, Andrew Y Ng, and Michael I Jordan. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993–1022, 2003 David M Blei and John D Lafferty. Topic models. Taylor and Francis, 2009. Ali Daud, Juanzi Li, Lizhu Zhou, and Faqir Muhammad. Knowledge discovery through directed probabilistic topic models: a survey. Frontiers of Computer Science in China, 4(2):280–301,January 2010. Mark Steyvers and Tom Griffith. Probabilistic topic models. Latent Semantic Analysis: A Road to Meaning. Laurence Erlbaum, July 2006. LDA : where to learn more --- Surveys
54.
55. 如何得到LDA中的参数 --- 推断 每个词的主题指定概率 文档中主题的概率 LDA模型中最重要的计算任务是计算隐含变量的后验概率 变分法 抽样法 Variational Inference Gibbs Sampling Inference --- get important parameters in LDA
57. 推断方法 随机方法 (抽样) MCMC, Metropolis-Hasting, Gibbs, etc 计算量大,但相对精确 判定方法 (优化) Mean Field, Belief Propagation Variational Bayes, Expectation Propagation 计算量小,不精确,可以给出边界 Inference Methods : Comparison of two major methods
58. 变分推断 Variational Inference Variational≈ Optimization Variational≈ ConvexOptimization The basic idea of convexity-based variational inference is to make use of Jensen’s inequality to obtain an adjustable lower bound on the log likelihood. Essentially, one considers a family of lower bounds, indexed by a set of variational parameters. The variationalparameters are chosen by an optimizationprocedure that attempts to find the tightest possible lower bound. Variational Inference
59. Mean field 基本思想 用一个简单可分解的分布 逼近 求KL散度最小的逼近 为何得名? 概率可完全分解 Mean field variational inference
66. 变分推断: 获得更多 Martin Wainwright. Graphical models and variational methods: Message-passing, convex relaxations, and all that. ICML2008 Tutorial M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, Vol. 1, Numbers 1--2, pp. 1--305, December 2008 Variational Inference : Where to learn more
67.
68. MCMC in LDA MCMC Overview Sampling in general Why sampling is necessary and why it is hard Importance sampling, rejection sampling Markov Chain Monte Carlo Metropolis-Hasting, Gibbs sampling Collapsed Gibbs in LDA
69. Pioneers to push sampling Nicholas C. Metropolis Andrey Markov Josiah W. Gibbs MCMC Overview
70. 抽样例子 中华人民共和国国家统计局 2006年3月16日 经国务院批准,我国于2005年底开展了全国1%人口抽样调查工作。这次调查的样本量为1705万人,占全国总人口的1.31%。 全国人口中,具有大学程度(指大专及以上)的人口为6764万人,高中程度(含中专)的人口为15083万人,初中程度的人口为46735万人,小学程度的人口为40706万人。 Sampling Example : Population statistics
74. Importance sampling In Rejection sampling, throwing away an x seems a waste, and the rejection is the only thing we know about the original distribution. Sampling : Importance sampling
78. Gibbs Sampling in LDA : Joint distribution 将上式带入 Collapsed: Gibbs Sampling in LDA
79. Gibbs Sampling in LDA : Joint distribution 此处省略若干公式 …… Gibbs Sampling in LDA
80. Gibbs Sampling in LDA : Marginal dist. 此处省略若干公式 …… Gibbs Sampling in LDA
81. Gibbs Sampling in LDA in Python Code Really simple! for m in xrange(n_docs): for i, w in enumerate(word_indices(matrix[m, :])): z = np.random.randint(self.n_topics) self.nmz[m,z] += 1 self.nm[m] += 1 self.nzw[z,w] += 1 self.nz[z] += 1 self.topics[(m,i)] = z Sampling : Gibbs Sampling code in Python
82. Gibbs Sampling获得更多 D.J.C. MacKay. Information theory, inference, and learning algorithms. Cambridge UnivPr,2003. GregorHeinrich. Parameter estimation for text analysis. Technical Report, 2009. Michael I. Jordan and Yair Weiss. Graphical models: Probabilistic inference. Christophe Andrieu, N De Freitas, A Doucet, and Michael I. Jordan. An introduction to MCMC for machine learning. Machine learning, pages 5–43, 2003. Yi Wang. Distributed Gibbs Sampling of Latent Dirichlet Allocation : The Gritty Details. Technical Report, 2007. Gibbs Sampling where to learn more
87. Correlated + Dynamic TM David M. Blei and John D Lafferty. Correlated Topic Models. In Advances in Neural Information Processing Systems 18, 2006. David M. Blei and John D Lafferty. A correlated topic model of Science. The Annals of Applied Statistics, 1(1):17–35, 2007. David M. Blei and John D Lafferty. Dynamic topic models. Proceedings of the 23rd international conference on Machine learning - ICML ’06, pages 113–120, 2006. Correlated + Dynamic Topic Models
93. Dynamic TM Dynamic Topic Models : Top10 words of Science and example articles of Science
94. Topics over time Published in: KDD '06 Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining Topics over time
104. Supervised LDA David Blei and Jon D. McAuliffe. Supervised topic models. In Advances in Neural Information Processing Systems, pages 1–22, 2008. Daniel Ramage, David Hall, Ramesh Nallapati, and C.D. Manning. Labeled LDA : A supervised topic model for credit attribution in multi-labeled corpora. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1, pages 248–256. Association for Computational Linguistics, 2009. Jun Zhu, Amr Ahmed, and Eric Xing. MedLDA: Maximum Margin Supervised Topic Models. Journal of Machine Learning Research, 1:1–48, 2010. Supervised LDA
130. 构件挖掘 12,151Java projects from Sourceforge and Apache 4,632projects 366,287source files 38.7 million lines of code written by 9,250developers Software Analysis with Unsupervised Topic Models
147. D. Blei and M. Jordan. Variational inference for Dirichlet process mixtures. Journal of Bayesian Analysis, 1:121–144, 2006.
148. M. Steyvers and T. Griffiths. Probabilistic Topic Models. In Latent Semantic Analysis: A Road to Meaning, T. Landauer, Mcnamara, S. Dennis, and W. Kintsch eds. Laurence Erlbaum, 2006.
149. Y. Teh, M. Jordan, M. Beal, and D. Blei. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101:1566-1581, 2006.
150. J. Zhu, A. Ahmed and E. P. Xing. MedLDA: Maximum Margin Supervised Topic Models for Regression and Classification. The 26th International Conference on Machine Learningy, 2009.Topic Models Background
151.
152. W. Li, and A. McCallum. Pachinko Allocation: DAG-structured Mixture Models of Topic Correlations. In International Conference on Machine Learning, 2006.
153. Porteous, A. Ascuncion, D. Newman, A. Ihler, P. Smyth, and M. Welling. Fast Collapsed Gibbs Sampling For Latent Dirichlet Allocation. In Knowledge Discovery and Data Mining, 2008.
154. H. Wallach, I. Murray, R. Salakhutdinov and D. Mimno. Evaluation Methods for Topic Models. In International Conference on Machine Learning, 2009.
155. M. Welling, Y. Teh and B. Kappen. Hybrid Variational/Gibbs Inference in Topic Models. In Uncertainty in Artificial Intelligence, 2008.Topic Models Background
156.
157. P. Agius, Y. Ying, and C. Campbell. Bayesian Unsupervised Learning with Multiple Data Types. Statistical Applications in Genetic and Molecular Biology, 3(1):27, 2009.
158. P. Flaherty, G. Giaever, J. Kumm, Michael I. Jordan, Adam P. Arkin. A Latent Variable Model for Chemogenomic Profiling. Bioinformatics 2005 Aug 1;21(15):3286-93.
159. S. Shringarpure and E. P. Xing. mStruct: Inference of Population Structure in Light of Both Genetic Admixing and Allele Mutations. Genetics, Vol 182, issue 2, 2009.Topic Models Background
160.
161. T. Griffiths, M. Steyvers, D. Blei, and J. Tenenbaum. Integrating topics and syntax. In Neural Information Processing Systems, 2005.
162. K. Toutanova and M. Johnson. A Bayesian LDA-based Model for Semi-Supervised Part-of-speech Tagging. In Neural Information Processing Systems, 2008. Topic Models Background
163.
164. L. Dietz, S. Bickel, and T. Scheffer. Unsupervised Prediction of Citation Influences. In International Conference on Machine Learning, 2007.
165. D. Hall, D. Jurafsky, and C. Manning. Studying the History of Ideas Using Topic Models. In Emperical Methods in Natural Language Processing, 2008.Topic Models Background
166.
167. J. Chang and D. Blei. Relational topic models for document networks. Artificial Intelligence and Statistics (in print), 2009.
168. E.P. Xing, W. Fu, and L. Song. A State-Space Mixed Membership Blockmodel for Dynamic Network Tomography. Annals of Applied Statistics, 2009.
169. H. Wallach. Topic Modeling: Beyond Bag-of-Words. In International Conference on Machine Learning, 2006. Topic Models Background
170.
171. L. Fei-Fei, R. Fergus and P. Perona. Learning generative visual models for 101 object categories. In Computer Vision and Image Understanding, 2007.
172. C. Wang, D. Blei and L. Fei-Fei. Simultaneous Image Classification and Annotation. In Computer Vision and Pattern Recognition, 2009. Topic Models Background