This document discusses topic models, LDA, and related concepts. It begins with an overview of LDA and how it uses a graphical model approach for unsupervised learning. Various inference methods for LDA are discussed, including variational inference and Gibbs sampling. The document also covers extensions like correlated topic models and dynamic topic models, as well as applications and researchers in the field. Key concepts covered include posterior approximation, sampling, variational methods, and optimization.

1. Topic Models, LDA and all that 肖智博 2011-03-23

2. LDA Linear Discriminant Analysis Fisher Linear Discriminant Analysis 有监督学习 寻找使类间与类内比最大的投影方向 从矩阵的角度 Latent DirichletAllocation 无监督学习 从图模型的角度 Just make sure we are on the same page

3. 所需数学知识 Latent Dirichlet Allocation 后验概率逼近方法 主题模型的演化 —— 从主题的关系探讨 有监督LDA --- MedLDA 主题模型的应用 主要内容 Roadmap

4. 后验概率 posteriori 逼近 approximation 采样 sampling 变分法variational 优化 optimization 关键词 Keywords

5. Approximation methods is useful! GibbsSampling Variational Methods (Convex) Optimization is Useful! Math is almighty! 我的感受 My afterthoughts

7. 概率知识回顾 Beta和Gamma方程 Dirichlet分布 多项分布 共轭分布 贝叶斯网络简介 本节内容 Overview

8. 概率知识复习 Chain rule (conditional independence) Bayes rule Marginal distribution Probability Recap

12. 期望

13. 方差Multinomial Distribution

14. 共轭分布 Very Important! 如果似然函数 和先验分布 属于同一分布族，则称两者是共轭分布 共轭先验分布可以为计算后验分布提供方便 Conjugrate distribution

15. 共轭分布 相似 Conjugrate distribution

17. David Barber: Bayesian Reasoning and Machine Learning

18. 贝叶斯网络 Bayesian Network

19. 贝叶斯网络(续) Bayesian Network

20. 贝叶斯网络：要解决的问题 如何表示满足特定独立性的分布？ 表示问题 representation 如何利用特定独立性来有效的计算？ 推断问题 inference 如何辨识数据中的特定独立性？ 学习问题 learning Bayesian Network : problems to solve

21. David Barber Bayesian Reasoning and Machine Learning Daphne Koller，NirFriedman Probabilistic Graphical Model Bishop Pattern Recognition and Machine Learning Ch8 Eric Xing Probabilistic Graphical Models 获得更多 Bayesian Network : where to learn more

23. 本节内容:主题模型 主题模型 LDA 推断方法 Inference 主题间关系 MedLDA LDA的应用 Topic Model Overview

25. Hyperspace Analogue to Language (HAL) (Lund and Burgess, 1996)

26. Bound Encoding of the Aggregate Language Environment(BEAGLE) (Jones and Mewhort, 2007)Topic Model Overview

28. 主要研究者 Mark Steyvers Michael I. Jordan Andrew Ng D Blei Andrew McCallum John Lafferty Eric Xing Fei-Fei Li Researchers in Topic Model

29. 主要研究者 Hanna Wallach Yee WhyeTeh Jun Zhu David Mimno Researchers in Topic Model

30. Why Latent?

32. 一个恰当的先验分布应该避免给可能发生的情况赋予小概率，但是也不应该将几乎不可能事件与其他事件一概而论。为了避免这种情况发生，需要考虑模型参数间的联系。一种策略是在模型中引入隐含变量(latent variables )，另一种是引入超参数(hyperparameters)。这两种方法都是可计算的(tractable)。 From Radford Neal’s CSC2541 “Bayesian Methods for Machine Learning”

33. Goal The goal is to find short descriptions of the members of a collection that enable efficient processing of large collections while preserving the essential statistical relationships that are useful for basic tasks such as classification, novelty detection, summarization, and similarity and relevance judgments. Goal and Motivation of Topic Model

34. 主题模型：前人工作 tf-idf 统计词频 无法捕捉到文档内部和文档间的统计特征 tf-idf 1983 Previous Work : tf-idf

35. 主题模型：前人工作 LSI: Latent Semantic Indexing 在词与文档(term-by-document)矩阵上使用SVD tf-idf的线性组合，能捕捉到一些语法特征 tf-idf 1983 LSI 1990 Previous work: LSI

36. 主题模型：前人工作 pLSI (aka Aspect Model 内容模型) 参数随着语料库的容量增长，容易过拟合 在文档层面没有一个统计模型，无法对文档指定概率 tf-idf 1983 pLSI 1999 LSI 1990 Previous work: pLSI

37. 主题模型：前人工作 LDA bag-of-word假设 同时考虑词和文档交换性的混合模型 tf-idf 1983 pLSI 1999 LSI 1990 LDA 2003 LDA

38. LDA LDA in graphical model

39. LDA举例：在线音乐社区 An analog : Modeling Shared Tastes in Online Communities - Laura Dietz NIPS 09 workshop

40. LDA 对于语料库 中的每个文档 ，LDA是如下的变参数层次贝叶斯网络： 选择单词的个数 选择文档中话题比率 对于每个单词 选择话题 从分布 中选择单词 LDA procedure

41. LDA 在已知超参数 和 的情况下，主题和词的联合概率为 对 和 求积分，可以得到文档的边际概率 进而，对所有的边际概率求积，可得语料库的概率 LDA : to see a document

42. LDA 在已知超参数 和 的情况下，主题和词的联合概率为 为何求积分？ 对 和 求积分，可以得到文档的边际概率 进而，对所有的边际概率求积，可得语料库的概率 LDA : to see a document

44. 数据是随机的，所以，期望也是随机的；

46. 求参数𝜃的期望是通过求其概率分布得到；

47. 对未知参数的估计是通过求其边际概率得到。 From Jerry Zhu’s CS 731 Advanced Artificial Intelligence

49. 段落的中心思想

50. 展开LDA by Human and Computer

51. LDA : Topic Five topics from a 50-topic LDA model fit to Science from 1980– 2002 LDA : Five topics from a 50-topic LDA model fit to Science from 1980– 2002

52. LDA : Personas Demo http://personas.media.mit.edu/personasWeb.html LDA : Personas

53. LDA ：获得更多 David M Blei, Andrew Y Ng, and Michael I Jordan. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993–1022, 2003 David M Blei and John D Lafferty. Topic models. Taylor and Francis, 2009. Ali Daud, Juanzi Li, Lizhu Zhou, and Faqir Muhammad. Knowledge discovery through directed probabilistic topic models: a survey. Frontiers of Computer Science in China, 4(2):280–301,January 2010. Mark Steyvers and Tom Griffith. Probabilistic topic models. Latent Semantic Analysis: A Road to Meaning. Laurence Erlbaum, July 2006. LDA : where to learn more --- Surveys

55. 如何得到LDA中的参数 --- 推断 每个词的主题指定概率 文档中主题的概率 LDA模型中最重要的计算任务是计算隐含变量的后验概率 变分法 抽样法 Variational Inference Gibbs Sampling Inference --- get important parameters in LDA

56. 推断方法 Inference Methods Overview

57. 推断方法 随机方法 (抽样) MCMC, Metropolis-Hasting, Gibbs, etc 计算量大，但相对精确 判定方法 (优化) Mean Field, Belief Propagation Variational Bayes, Expectation Propagation 计算量小，不精确，可以给出边界 Inference Methods : Comparison of two major methods

58. 变分推断 Variational Inference Variational≈ Optimization Variational≈ ConvexOptimization The basic idea of convexity-based variational inference is to make use of Jensen’s inequality to obtain an adjustable lower bound on the log likelihood. Essentially, one considers a family of lower bounds, indexed by a set of variational parameters. The variationalparameters are chosen by an optimizationprocedure that attempts to find the tightest possible lower bound. Variational Inference

59. Mean field 基本思想 用一个简单可分解的分布 逼近 求KL散度最小的逼近 为何得名？ 概率可完全分解 Mean field variational inference

60. LDA中的变分推断 目标：求出 Variational inference in LDA Overview

61. LDA中的变分推断 Jensen不等式 Variational Inference : Beautiful math

62. LDA中的变分推断 记 则 因为 都是可分解的，所以有 Variational Inference : Beautiful math

63. LDA中的变分推断 Variational Inference : Beautiful math

64. LDA中的变分推断 应用拉格朗日法，得到 Variational Inference : Beautiful math

65. 总结：LDA中的变分推断 目标：求出 Variational Inference : Review

66. 变分推断: 获得更多 Martin Wainwright. Graphical models and variational methods: Message-passing, convex relaxations, and all that. ICML2008 Tutorial M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, Vol. 1, Numbers 1--2, pp. 1--305, December 2008 Variational Inference : Where to learn more

68. MCMC in LDA MCMC Overview Sampling in general Why sampling is necessary and why it is hard Importance sampling, rejection sampling Markov Chain Monte Carlo Metropolis-Hasting, Gibbs sampling Collapsed Gibbs in LDA

69. Pioneers to push sampling Nicholas C. Metropolis Andrey Markov Josiah W. Gibbs MCMC Overview

70. 抽样例子 中华人民共和国国家统计局 2006年3月16日 经国务院批准，我国于2005年底开展了全国1%人口抽样调查工作。这次调查的样本量为1705万人，占全国总人口的1.31%。 全国人口中，具有大学程度（指大专及以上）的人口为6764万人，高中程度（含中专）的人口为15083万人，初中程度的人口为46735万人，小学程度的人口为40706万人。 Sampling Example : Population statistics

71. 抽样要解决的问题 1 从给定概率分布 中产生样本 2 在给定概率分布 下，估计函数的期望 Sampling

72. 例子：测量湖水内某种物质的含量 Sampling ： Why it is so damn hard?

73. Rejection sampling Reject Accept Sampling ： Rejection sampling

74. Importance sampling In Rejection sampling, throwing away an x seems a waste, and the rejection is the only thing we know about the original distribution. Sampling ： Importance sampling

75. Metropolis-Hasting Method 考虑Markov特性：某个状态仅与其前一个状态有关 在t状态， 可以是任意可以抽样的分布，比如高斯分布 对于一个新的状态，考虑 Sampling : Metropolis-Hasting Method

76. Gibbs Sampling 1 对所有的变量初始化 2 选定维度 i 从分布 中对 采样 Sampling : Gibbs Sampling

77. Gibbs Sampling in LDA : Joint distribution Gibbs Sampling in LDA

78. Gibbs Sampling in LDA : Joint distribution 将上式带入 Collapsed： Gibbs Sampling in LDA

79. Gibbs Sampling in LDA : Joint distribution 此处省略若干公式 …… Gibbs Sampling in LDA

80. Gibbs Sampling in LDA : Marginal dist. 此处省略若干公式 …… Gibbs Sampling in LDA

81. Gibbs Sampling in LDA in Python Code Really simple! for m in xrange(n_docs): for i, w in enumerate(word_indices(matrix[m, :])): z = np.random.randint(self.n_topics) self.nmz[m,z] += 1 self.nm[m] += 1 self.nzw[z,w] += 1 self.nz[z] += 1 self.topics[(m,i)] = z Sampling : Gibbs Sampling code in Python

82. Gibbs Sampling获得更多 D.J.C. MacKay. Information theory, inference, and learning algorithms. Cambridge UnivPr,2003. GregorHeinrich. Parameter estimation for text analysis. Technical Report, 2009. Michael I. Jordan and Yair Weiss. Graphical models: Probabilistic inference. Christophe Andrieu, N De Freitas, A Doucet, and Michael I. Jordan. An introduction to MCMC for machine learning. Machine learning, pages 5–43, 2003. Yi Wang. Distributed Gibbs Sampling of Latent Dirichlet Allocation : The Gritty Details. Technical Report, 2007. Gibbs Sampling where to learn more

85. Dynamic topic models

86. Temporal topic modelsEvolution of Topic Models

87. Correlated + Dynamic TM David M. Blei and John D Lafferty. Correlated Topic Models. In Advances in Neural Information Processing Systems 18, 2006. David M. Blei and John D Lafferty. A correlated topic model of Science. The Annals of Applied Statistics, 1(1):17–35, 2007. David M. Blei and John D Lafferty. Dynamic topic models. Proceedings of the 23rd international conference on Machine learning - ICML ’06, pages 113–120, 2006. Correlated + Dynamic Topic Models

88. Correlated + Dynamic TM Correlated + Dynamic Topic Models

89. Correlated TM 无法捕捉主题间的联系 和多元分布不共轭 采用变分法进行推断 Correlated Topic Models

90. Correlated TM Correlated Topic Models

91. Correlated TM 控制稀疏度 Correlated Topic Models

92. Dynamic TM DTM中，假设所有文档是按时间分块 Dynamic Topic Models

93. Dynamic TM Dynamic Topic Models ： Top10 words of Science and example articles of Science

94. Topics over time Published in: KDD '06 Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining Topics over time

95. Topics over time Topics over time

96. Topics over time : Topic discovery TOT LDA Topics over time : State-of-the-Union Addresses

97. Topics over time : Topic evolution Topics over time : Topic evolution on NIPS data

98. Topics over time : Co-occuring Topics Topics over time : Topic evolution on NIPS data

99. Topics over time : Review Topics over time : Review 基于LDA 话题演化研究方法综述@中文信息学报 2010 年11 月

100. Work by Hanna Wallach NIPS ’09 ICML ’09 Work by Hanna Wallach

101. 不对称先验 Rethinking LDA : Why priors matter

103. 𝜃𝑑是每个文档的参数，适合使用不对称先验对文档进行刻画 Rethinking LDA : Why priors matter

104. Supervised LDA David Blei and Jon D. McAuliffe. Supervised topic models. In Advances in Neural Information Processing Systems, pages 1–22, 2008. Daniel Ramage, David Hall, Ramesh Nallapati, and C.D. Manning. Labeled LDA : A supervised topic model for credit attribution in multi-labeled corpora. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1, pages 248–256. Association for Computational Linguistics, 2009. Jun Zhu, Amr Ahmed, and Eric Xing. MedLDA: Maximum Margin Supervised Topic Models. Journal of Machine Learning Research, 1:1–48, 2010. Supervised LDA

105. Supervised LDA ：目标 Naïve 在无监督模型的基础上，增加对类标的描述 进行分类、回归 朴素的方法：先进行LDA，利用主题进行分类 用分布来对类标进行建模 Supervised LDA

106. Supervisedtopic models 用泛化线性模型(Generalized Linear Model)来对类标进行建模 Supervised LDA

108. 二项分布

109. 多项分布

110. 柏松分布

111. Gamma分布

112. Weibull分布

113. ……Supervised LDA : Why use GLM

114. Semi-Supervised LDA Unlabeled Labeled Semi-Supervised LDA

115. MedLDA maximum entropy discrimination latent Dirichlet allocation 通过优化单一目标函数和一组边界约束将大边界理论同主题模型结合在一起 Maximum Entropy Discrimination Latent Dirichlet Allocation

116. MedTM 优点： 利用大边界理论正确分类 更好的描述数据 Maximum Entropy Discrimination Topic Models

117. MedLDA : topic discovery MedLDA LDA 2D embedding on 20Newsgroups data

118. MedLDA : classification 二分类 多分类 Classfication on 20Newsgroups data

120. 文本摘要

121. 多媒体（图像、音频、视频）

122. 蛋白质表达式分析NIPS ’09 Workshop on Applications for Topic Models: Text and Beyond

123. 重建庞贝古城 Reconstructing Pompeian Households

124. 重建庞贝古城 Reconstructing Pompeian Households

125. 重建庞贝古城 炖菜锅 糕点模具 Reconstructing Pompeian Households

126. 重建庞贝古城 Is it so? 炖菜锅 糕点模具 Reconstructing Pompeian Households

127. 重建庞贝古城 房间 主题 功能 分布 物件 单词 Reconstructing Pompeian Households

128. 重建庞贝古城 Reconstructing Pompeian Households

129. 重建庞贝古城 物品用途与名称不匹配 Reconstructing Pompeian Households

130. 构件挖掘 12,151Java projects from Sourceforge and Apache 4,632projects 366,287source files 38.7 million lines of code written by 9,250developers Software Analysis with Unsupervised Topic Models

131. 构件挖掘 Software Analysis with Unsupervised Topic Models

132. 构件挖掘 Software Analysis with Unsupervised Topic Models

135. 以MedLDA为基础继续扩展

136. LDA+主动学习

137. 半监督主题模型

138. Asymmetric Dirichlet Prior

139. 更好的逼近方法(Data Augmentation)

140. 文本摘要

141. 非结构化数据的主题模型(微博、论坛、基因序列)

142. LDA应用到图像Future Direction

144. nested Chinese Restaurant Process

145. Indian Buffet ProcessNot covered topics

147. D. Blei and M. Jordan. Variational inference for Dirichlet process mixtures. Journal of Bayesian Analysis, 1:121–144, 2006.

148. M. Steyvers and T. Griffiths. Probabilistic Topic Models. In Latent Semantic Analysis: A Road to Meaning, T. Landauer, Mcnamara, S. Dennis, and W. Kintsch eds. Laurence Erlbaum, 2006.

149. Y. Teh, M. Jordan, M. Beal, and D. Blei. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101:1566-1581, 2006.

150. J. Zhu, A. Ahmed and E. P. Xing. MedLDA: Maximum Margin Supervised Topic Models for Regression and Classification. The 26th International Conference on Machine Learningy, 2009.Topic Models Background

152. W. Li, and A. McCallum. Pachinko Allocation: DAG-structured Mixture Models of Topic Correlations. In International Conference on Machine Learning, 2006.

153. Porteous, A. Ascuncion, D. Newman, A. Ihler, P. Smyth, and M. Welling. Fast Collapsed Gibbs Sampling For Latent Dirichlet Allocation. In Knowledge Discovery and Data Mining, 2008.

154. H. Wallach, I. Murray, R. Salakhutdinov and D. Mimno. Evaluation Methods for Topic Models. In International Conference on Machine Learning, 2009.

155. M. Welling, Y. Teh and B. Kappen. Hybrid Variational/Gibbs Inference in Topic Models. In Uncertainty in Artificial Intelligence, 2008.Topic Models Background

157. P. Agius, Y. Ying, and C. Campbell. Bayesian Unsupervised Learning with Multiple Data Types. Statistical Applications in Genetic and Molecular Biology, 3(1):27, 2009.

158. P. Flaherty, G. Giaever, J. Kumm, Michael I. Jordan, Adam P. Arkin. A Latent Variable Model for Chemogenomic Profiling. Bioinformatics 2005 Aug 1;21(15):3286-93.

159. S. Shringarpure and E. P. Xing. mStruct: Inference of Population Structure in Light of Both Genetic Admixing and Allele Mutations. Genetics, Vol 182, issue 2, 2009.Topic Models Background

161. T. Griffiths, M. Steyvers, D. Blei, and J. Tenenbaum. Integrating topics and syntax. In Neural Information Processing Systems, 2005.

162. K. Toutanova and M. Johnson. A Bayesian LDA-based Model for Semi-Supervised Part-of-speech Tagging. In Neural Information Processing Systems, 2008. Topic Models Background

164. L. Dietz, S. Bickel, and T. Scheffer. Unsupervised Prediction of Citation Influences. In International Conference on Machine Learning, 2007.

165. D. Hall, D. Jurafsky, and C. Manning. Studying the History of Ideas Using Topic Models. In Emperical Methods in Natural Language Processing, 2008.Topic Models Background

167. J. Chang and D. Blei. Relational topic models for document networks. Artificial Intelligence and Statistics (in print), 2009.

168. E.P. Xing, W. Fu, and L. Song. A State-Space Mixed Membership Blockmodel for Dynamic Network Tomography. Annals of Applied Statistics, 2009.

169. H. Wallach. Topic Modeling: Beyond Bag-of-Words. In International Conference on Machine Learning, 2006. Topic Models Background

171. L. Fei-Fei, R. Fergus and P. Perona. Learning generative visual models for 101 object categories. In Computer Vision and Image Understanding, 2007.

172. C. Wang, D. Blei and L. Fei-Fei. Simultaneous Image Classification and Annotation. In Computer Vision and Pattern Recognition, 2009. Topic Models Background

173. One more thing…

174. 房间 主题 关系 结构 功能 分布 物件 单词 广泛应用

176. Q & A