Used Overview’s “topic tree” (TF-IDF clustering) to find a group of key
emails from a listserv. Summer 2013.
Technical troubles with a new system meant that almost 70,000 North
Carolina residents received their food stamps late this summer. That’s 8.5
percent of the number of clients the state currently serves every month. The
problem was eventually traced to web browser compatibility issues. WRAL
reporter Tyler Dukes obtained 4,500 pages of emails — on paper — from
various government departments and used DocumentCloud and Overview to
piece together this story.
Finalist, 2014 Pulitzer Prize in Public Service
Handcrafted list of source URLs (news front pages) and links followed
to depth 4
“For each page examined, if the amount of text in the largest cell of the
page (after stripping tags and links) is greater than some particular
constant (currently 512 characters), it is assumed to be a news article,
and this text is extracted.”
(At least it’s simple. This was 2002. How often does this work now?)
Text extraction from HTML
Now multiple services/apis to do this, e.g. Mercury
• encode articles into feature vectors
• cosine distance function
• hierarchical clustering algorithm
But news is an on-line problem...
Articles arrive one at a time, and must be clustered immediately.
Can’t look forward in time, can’t go back and reassign.
Single pass clustering
put first story in its own cluster
get next story S
look for cluster C with distance < T
put S in C
put S in new cluster
Now sort events into categories
U.S., World, Finance, Science and Technology, Entertainment, Sports.
Primitive operation: what topic is this story in?
Each category has pre-assigned TF-IDF coordinate. Story category =
Problem: given a set of documents, write a sentence summarizing
Active research area in AI. Recent progress with recurrent neural
network techniques. But initial algorithms go back to 1950s.
Generating News Headlines
with Recurrent Neural
We used a machine-learning method
known as latent Dirichlet allocation to
identify the topics in all 14,400 petitions
and to then categorize the briefs. This
enabled us to identify which lawyers did
which kind of work for which sorts of
petitioners. For example, in cases where
workers sue their employers, the lawyers
most successful getting cases before the
court were far more likely to represent the
employers rather than the employees.
The Echo Chamber, Reuters
Can the computer tell us the “topics” in a document set? Can the computer
organize the documents by “topic”?
Note: TF-IDF tells us the topics of a single document, but here we want topics
of an entire document set.
One simple technique
Sum TF-IDF scores for each word across entire document set, choose top
Cluster descriptions in Overview prototype generated this way
Topic Modeling Algorithms
Basic idea: reduce dimensionality of document vector space, so each
dimension is a topic.
Each document is then a vector of topic weights. We want to figure out what
dimensions and weights give a good approximation of the full set of words in
Many variants: LSI, PLSI, LDA, NMF
Approximate term-document matrix V as product of two
lower rank matrixes
m docs by n terms m docs by r "topics" r "topics" by n terms
A "topic" is a group of words that occur together.
words in this topic
topics in this document
Non-negative Matrix Factorization
All elements of document coordinate matrix W and topic matrix H
must be >= 0
Simple iterative algorithm to compute.
Still have to choose number of topics r
Latent Dirichlet Allocation
Imagine that each document is written by someone going through the
1. For each doc d, choose mixture of topics p(z|d)
2. For each word w in d, choose a topic z from p(z|d)
3. Then choose word from p(w|z)
A document has a distribution of topics.
Each topic is a distribution of words.
LDA tries to find these two sets of distributions.
LDA models each document as a distribution over topics. Each word
belongs to a single topic.
LDA models a topic as a distribution over all the words in the corpus. In
each topic, some words are more likely, some are less likely.
topic for word word in doc
topics in doc
LDA Plate Notation
words in topics
word[d][i] document words
k # topics
a doc topic concentration
b topic word concentration
n # docs
len[d] # words in document
v vocabulary size
topics -> topic_words
topic_words[*][*] = b
topic_words[topics[d][i]][word[d][i]] += 1
topics -> doc_topics
doc_topics[*][*] = a
// for each word in document, sample a new topic
w = word[d][i]
p[t] = doc_topics[d][j] * topic_words[j][w]
topics[d][i] = sample from p[t]
Output of NMF and LDA is a vector of much lower dimension for each
document. ("Document coordinates in topic space.")
Dimensions are “concepts” or “topics” instead of words.
Can measure cosine distance, cluster, etc. in this new space.
More than a Million Pro-Repeal Net Neutrality Comments were Likely Faked, Jeff
Word vectors with semantics
Word embedding is a function f(w) which maps a string (word) to a vector
with ~200 dimensions.
We hope that words with similar meanings have similar vectors. Then we
can do things like:
- Compare the meaning of two words
- Compare the meaning of two sentences
- Classify words/sentences by topic
- Implement semantic search
…basically everything we use TF-IDF for.
The distributional hypothesis in linguistics is derived from the semantic
theory of language usage, i.e. words that are used and occur in the same
contexts tend to have similar meanings. The underlying idea that "a word
is characterized by the company it keeps" was popularized by Firth.
Google ngram viewer - 12% of all English books
Comparing two document sets
Let me talk about Downton Abbey for a minute. The show's popularity has
led many nitpickers to draft up lists of mistakes. ... But all of these have
relied, so far as I can tell, on finding a phrase or two that sounds a bit off,
and checking the online sources for earliest use.
I lack such social graces. So I thought: why not just check every single line
in the show for historical accuracy? ... So I found some copies of the
Downton Abbey scripts online, and fed every single two-word phrase
through the Google Ngram database to see how characteristic of the
English Language, c. 1917, Downton Abbey really is.
- Ben Schmidt, Making Downton more traditional
Bigrams that do not appear in English books between 1912 and
Bigrams that are at least 100 times more common today than
they were in 1912-1921
What is this investigative journalist doing with documents?
A number of previous tools aim to help the user “explore” a
document collection (such as [6, 9, 10, 12]), though few of
these tools have been evaluated with users from a specific
target domain who bring their own data, making us suspect
that this imprecise term often masks a lack of understanding
of actual user tasks.
Overview: The Design, Adoption, and Analysis of a Visual Document
Mining Tool For Investigative Journalists, Brehmer et al, 2014
What do Journalists do with Documents, Stray 2016
What are the challenges
that standard NLP doesn’t address?
Muckrock.com can automate some parts of the FOIA request process.
Getting data is hard
Data is insanely contextual
VICTS AND SUSPS BECAME INV IN VERBA ARGUMENT SUSP
THEN BEGAN HITTING VICTS IN THE FACE
Typical incident description processed by LA Times crime classifier
If your algorithm isn’t robust to input noise, don’t even bother
Data is insanely dirty – OCR and more
Most NER implementations have low recall (~70%)
Investigations would prefer higher recall, lower precision
out of 150
Entity recognition is not solved!
Entity recognition that depends on parsing fails.
Suffolk County public safety committee transcript,
Reference to a body left on the street due to union dispute
(via Adam Playford, Newsday, 2014)
Text search may not find the target
Journalists have problems
well beyond state of the art of NLP
A document describing a modification to one of the loans used to finance the Trump Soho
hotel (New York City ACRIS document 2006083000784001)
Excerpt of the hand-built chronological list of New York City real estate public records
concerning the Trump Soho hotel. Color coding indicates documents on the same date
(Giannina Segnini / Columbia Journalism School)