[Kim+ ICML2012] Dirichlet Process with Mixed Random Measures : A Nonparametric Topic Model for Labeled Data
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

[Kim+ ICML2012] Dirichlet Process with Mixed Random Measures : A Nonparametric Topic Model for Labeled Data

  • 19,837 views
Uploaded on

Supervised Nonparametric Topic Model

Supervised Nonparametric Topic Model

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
19,837
On Slideshare
2,239
From Embeds
17,598
Number of Embeds
19

Actions

Shares
Downloads
45
Comments
0
Likes
1

Embeds 17,598

http://shuyo.wordpress.com 10,454
http://d.hatena.ne.jp 6,632
https://shuyo.wordpress.com 272
http://hatenatunnel.appspot.com 160
http://webcache.googleusercontent.com 25
http://www.hatenatunnel.appspot.com 11
http://cloud.feedly.com 9
https://si0.twimg.com 7
http://translate.googleusercontent.com 5
http://prlog.ru 4
http://digg.com 3
http://208.71.46.190 3
https://twimg0-a.akamaihd.net 3
https://twitter.com 2
http://feedly.com 2
http://www.365dailyjournal.com 2
https://www.google.co.jp 2
http://www.google.com 1
http://www.google.ca 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. [Kim+ ICML2012] Dirichlet Process with Mixed Random Measures : A Nonparametric Topic Model for Labeled Data 2012/07/28 Nakatani Shuyo @ Cybozu Labs, Inc twitter : @shuyo
  • 2. LDA(Latent Dirichlet Allocation) [Blei+ 03]• Unsupervised Topic Model – Each word has an unobserved topic• Parametric – The topic size K is given in advance via Wikipedia
  • 3. Labeled LDA [Ramage+ 09]• Supervised Topic Model – Each document has an observed label• Parametric via [Ramage+ 09]
  • 4. Generative Process for L-LDA• 𝜷 𝑘 ~Dir 𝜼 topics corresponding to 𝑑 observed labels• Λ 𝑘 ~Bernoulli Φ 𝑘• 𝜽 𝑑 ~Dir 𝜶 𝑑 restricted to labeled – where 𝜶 𝑑 = 𝛼𝑘 parameters 𝑑 𝑘 Λ 𝑘 =1 𝑑 𝑑• 𝑧 𝑖 ~Multi 𝜽 𝑑• 𝑤𝑖 ~Multi 𝜷 𝑧 𝑑 𝑖 via [Ramage+ 09]
  • 5. Pros/Cons of L-LDA• Pros – Easy to implement• Cons via [Ramage+ 09] – It is necessary to specify label-topic correspondence manually • Its performance depends on the corresponds ※) My implementation is here : https://github.com/shuyo/iir/blob/master/lda/llda.py
  • 6. DP-MRM [Kim+ 12] – Dirichlet Process with Mixed Random Measures• Supervised Topic Model• Nonparametric – K is not the topic size, but the label size 𝛼 𝑁𝑗 𝐻 𝐺0𝑘 𝐺𝑗 𝜃 𝑗𝑖 𝑥 𝑗𝑖 𝜆j 𝑟𝑗 𝐷 𝛽 𝛾𝑘 𝜂 𝐾
  • 7. Generative Process for DP-MRM 𝛼 Each label has a random measure as topic space 𝑁𝑗 𝐻 𝐺0𝑘 𝐺𝑗 𝜃 𝑗𝑖 𝑥 𝑗𝑖• 𝐻 = Dir 𝛽 𝜆j 𝑟𝑗 𝐷• 𝐺0𝑘 ~DP 𝛾 𝑘 , 𝐻 𝛽 𝐾 𝛾𝑘 𝜂• 𝜆 𝑗 ~Dir 𝒓 𝑗 𝜂 where 𝒓 𝑗 = 𝐼 𝑘∈label 𝑗• 𝐺 𝑗 ~DP 𝛼, 𝑘∈label 𝑗 𝜆 𝑗𝑘 𝐺0𝑘 mixed random measures• 𝜃 𝑗𝑖 ~𝐺 𝑗 , 𝑥 𝑗𝑖 ~𝐹 𝜃 𝑗𝑖 = Multi 𝜃 𝑗𝑖
  • 8. Stick Breaking Process• 𝑣 𝑙 𝑘 ~Beta 1, 𝛾 𝑘 , 𝜋 𝑙𝑘 = 𝑣 𝑙 𝑘 𝑙−1 𝑑=0 1 − 𝑣 𝑑𝑘• 𝜙 𝑙𝑘 ~𝐻, 𝐺0𝑘 = ∞ 𝑙=0 𝜋 𝑙𝑘 𝛿 𝜙 𝑘 𝑙 𝑡−1• 𝜆 𝑗 ~Dir 𝒓 𝑗 𝜂 , 𝑤 𝑗𝑡 ~Beta 1, 𝛼 , 𝜋 𝑗𝑡 = 𝑤 𝑗𝑡 𝑑=0 1 − 𝑤 𝑗𝑑 𝑘 𝑗𝑡 ∞• 𝑘 𝑗𝑡 ~Multi 𝜆 𝑗 , 𝜓 𝑗𝑡 ~𝐺0 , 𝐺𝑗 = 𝑡=0 𝜋 𝑗𝑡 𝛿 𝜓 𝑗𝑡
  • 9. Chinese Restaurant Franchise• 𝑡 𝑗𝑖 : table index of 𝑖-th term in 𝑗-th document• 𝑘 𝑗𝑡 , 𝑙 𝑗𝑡 : dish indexes on 𝑡-th table of 𝑗-th document This layer consists on only a single DP G0 on normal HDP
  • 10. Inference (1)• Sampling 𝑡
  • 11. Inference (2)• Sampling 𝑘 and 𝑙
  • 12. Experiments• DP-MRM gives label-topic probabilistic corresponding automatically. via [Kim+ 12]
  • 13. via [Kim+ 12]• L-LDA can also predict single labeled document to assign a common second label to any documents.
  • 14. References• [Kim+ ICML2012] Dirichlet Process with Mixed Random Measures : A Nonparametric Topic Model for Labeled Data• [Ramage+ EMNLP2009] Labeled LDA : A supervised topic model for credit attribution in multi-labeled corpora• [Blei+ 2003] Latent Dirichlet Allocation