Successfully reported this slideshow.
Your SlideShare is downloading. ×

Dynamic Topic Modeling via Non-negative Matrix Factorization (Dr. Derek Greene)


Check these out next

1 of 24 Ad

More Related Content

Slideshows for you (17)

Viewers also liked (20)


Similar to Dynamic Topic Modeling via Non-negative Matrix Factorization (Dr. Derek Greene) (20)

More from Sebastian Ruder (20)


Recently uploaded (20)

Dynamic Topic Modeling via Non-negative Matrix Factorization (Dr. Derek Greene)

  1. 1. Dynamic Topic Modeling via Non-negative Matrix Factorization Derek Greene University College Dublin
  2. 2. Overview • Topic Modeling • Non-negative Matrix Factorization • Dynamic Topic Modeling • Proposed Approach • Dynamic Topic Modeling via Non-negative 
 Matrix Factorization • Application • Topic Modeling European 
 Parliamentary Speeches September 2016 2
  3. 3. Topic Modeling September 2016 3 • Goal: Discover hidden thematic structure in a corpus of text 
 (e.g. tweets, Facebook posts, news articles, political speeches). • Unsupervised approach, no prior annotation required. Input Output Data
 Preparation Topic Modeling Algorithm Topic 1 Topic 2 Topic k • Output of topic modeling is a set of k topics. Each topic has: 1. A descriptor, based on highest-ranked terms for the topic. 2. Membership weights for all documents relative to the topic.
  4. 4. Topic Modeling with NMF • Non-negative Matrix Factorization (NMF): Family of linear algebra algorithms for identifying the latent structure in data represented as a non-negative matrix (Lee & Seung, 1999). • NMF can be applied for topic modeling, where the input is a document-term matrix, typically TF-IDF normalized. September 2016 4 Input Matrix 
 (documents x terms) • Input: Document-term matrix A; User-specified number of topics k. • Output: Two k-dimensional factors W and H approximating A. An m Factor
 (documents x topics) NMF Wn k Factor
 (topics x terms) H m k·
  5. 5. Example: NMF Topic Modeling • Apply standard NMF to document-term matrix A (6 rows x 10 columns) for k=3 topics… September 2016 5 document 1 document 2 document 3 document 4 document 5 document 6 research stem education disease patient health budget finance banking bonds
  6. 6. Example: NMF Topic Modeling September 2016 6 research stem education disease patient health budget finance banking bonds Topic 1 Topic 2 Topic 3 Factor H
 Weights for terms document 1 document 2 document 3 document 4 document 5 document 6 Topic 1 Topic 2 Topic 3 Factor W 
 Weights for documents
  7. 7. (D. Blei, 2012) Dynamic Topic Models • Standard topic modeling approaches assume the order of documents does not matter. Not suitable for time-stamped data. • Dynamic topic modeling: Approaches to track how language changes and topics evolve over time in a time-stamped corpus. September 2016 7 Inaugural address
  8. 8. Dynamic Topic Modeling via Non-negative Matrix Factorization
  9. 9. Proposed Approach • Two-Level approach: Link together related topics found in different time windows to track topics over time. 9 Rank Term 1 eurozone 2 greece 3 imf 4 loan 5 debt Rank Term 1 greece 2 debt 3 germany 4 reparations 5 eu Rank Term 1 greece 2 russia 3 debt 4 eu 5 loan Topic in
 Window 1 Topic in
 Window 2 Topic in
 Window 3 Divide corpus into 𝜏 time windows of equal duration (e.g. days, weeks, months, quarters, or years). Level 1: Apply NMF topic modeling to documents in each window to produce window topics. Level 2: Apply another layer of NMF to all topics from Step 1 to find dynamic topics which span multiple time windows.
  10. 10. Proposed Approach • Key Idea for Level 2: • View the topic basis vectors (columns of factor H) found in each time window as “topic documents”. • Construct a new combined representation from these H factors. Similar to idea of “stacking” in supervised ensembles. • Apply NMF to this new representation. September 2016 10 𝜏 x Time Window 
 Datasets 𝜏 x NMF H Factors Factor H from Window 1 Factor H from Window 2 Factor H from Window 3 Factor H from Window 𝜏 … m’ terms n’topicdocuments Topic-Term Matrix
  11. 11. Example: Dynamic Topic Modeling 11 Topic-term matrix for 2 time window results, each with 3 topics. Window1-01 Window1-02 Window1-03 Window2-01 Window2-02 Window2-03 Topics for
 Window 1 Topics for
 Window 2 health patient disease citizen research education budget finance banking Topic-Term Matrix Heatmap
  12. 12. Application:
 European Parliament Collaboration with Dr. James Cross 
 UCD School of Politics & 
 International Relations
  13. 13. Exploring the European Parliament Agenda September 2016 13 • Directly elected parliamentary institution of the EU. • 8th term began in July 2014. • 751 Members of European Parliament (MEPs) from 28 member states. • 12 plenary sessions per year are held in Strasbourg. • During sessions, members may speak after being called by the President. Speaking time available to MEPs is strictly limited. • MEPs use speeches to state their positions on policies, to explain votes, and to demonstrate to their electorates that they are representing their interests in Europe.
  14. 14. Data Collection • In Autumn 2014 we collected ~400k records from EuroParl. • Covers activities of MEPS in the European parliament during terms 5-7 (1999-2014). • Focus on records of speeches in plenary. Accounts for 54.3% of all Europarl records. 14
  15. 15. Data Collection • Original corpus contains 269,696 plenary speeches. • Identified subset of 210,247 English language speeches, either native or translated. 15 • Divided these into 60 “time window” datasets. Each time window is a quarter from 1999-Q3 to 2014-Q2. Time Window (Quarter Number) NumberofSpeeches
  16. 16. Time Window Topic Modeling • Applied NMF to document-term matrix for the speeches in each of the 60 time windows. • Use automated topic coherence approach to choose number of topics k for each window (O’Callaghan et al, 2015). ➡ Output: 60 sets of time window topics. September 2016 16
  17. 17. Time Window Topic Modeling Example Topic: 2003-Q1 17 Top 10 terms suggest that this topic relates to the Iraq war. Top 10 speeches for this topic provide the context.
  18. 18. Dynamic Topic Modeling Results • Applying dynamic topic modeling to the resulting topic-term matrix with parameter selection yields 57 dynamic topics which show varied nature of European Parliament’s agenda… 18
  19. 19. Example: Climate Change 19 0 100 200 300 400 500 600 2000 2002 2004 2006 2008 2010 2012 2014 NumberofSpeeches Year Climate Change
 Package Cancun CopenhagenMontreal
  20. 20. Example: Financial & Euro Crisis 20 0 200 400 600 800 1000 1200 2000 2002 2004 2006 2008 2010 2012 2014 NumberofSpeeches Year Financial crisis Euro crisis A D C B
  21. 21. Dynamic Topics by Politician We associate MEPs with dynamic topics based on the number of speeches by the MEP associated with its window topics. September 2016 21 Pat Cox (Ireland) Top 10 Most Relevant Dynamic Topics
  22. 22. Dynamic Topics by Country 22 Ireland Cyprus
  23. 23. More Information European Parliament Speeches - Topic Explorer September 2016 23 Python Code and Documentation D. Greene, J. P. Cross, “Unveiling the Political Agenda of the European Parliament Plenary: A Topical Analysis,” in Proc. ACM Web Science’15, 2015. @derekgreene D. Greene, J. P. Cross. “Exploring the political agenda of the European parliament using a dynamic topic modeling approach”, Political Analysis, 2017 (in press).
  24. 24. References • D. Blei, A. Y. Ng, M. Jordan. “Latent dirichlet allocation”. Journal of Machine Learning Research, 3:993–1022, 2003. • D. Blei. “Probabilistic topic models”. Communications of the ACM, 2012. • D. D. Lee & H. S. Seung. “Learning the parts of objects by non-negative matrix factorization”. Nature, 401:788–91, 1999. • D. O’Callaghan, D. Greene, J. Carthy & P. Cunningham. “An analysis of the coherence of descriptors in topic modeling”. Expert Systems with Applications (ESWA), 2015. • Zhao, Wayne Xin, et al. "Comparing twitter and traditional media using topic models." Advances in Information Retrieval, 2011. • J. Grimmer. “A Bayesian Hierarchical Topic Model for Political Texts: Measuring Expressed Agendas in Senate Press Releases.” Political Analysis 18 (1). 1–35, 2010. September 2016 24