Data Science at The New York Times: what industry can learn from us; what we can learn from industry

data science @ The New York Times:
chris.wiggins@columbia.edu
chris.wiggins@nytimes.com
what industry can learn from us; what we can learn from industry

data science @ The New York Times

biology: 1892 vs. 1995
biology changed for good.

new toolset, new mindset

“These are indeed exciting times, not unlike the early days
of recombinant DNA in the 1970s, in which a
revolutionary new technology permitted entirely new
questions about the nature of genes to be raised. This
challenge is new to biology, and its resolution will require,
in addition to existing paradigms of molecular biology,
new sets of analytical tools... disciplines outside of biology
will be required to collaborate on this problem.”
new toolset, new mindset

genetics: 1837 vs. 2012
ML toolset; data science mindset

genetics: 1837 vs. 2012
ML toolset; data science mindset
arxiv.org/abs/1105.5821 ; github.com/rajanil/mkboost
Raj, Dewar, Palacios, Rabadan, CW

data science: mindset & toolset

drew conway, 2010

develop + deploy
machine learning solutions
to
newsroom + business problems

data science @ The New York Timesdata science @ The New York Times

news: 20th century
church state

news: 21st century
church state
data

learnings
- descriptive modeling
- predictive modeling
- prescriptive modeling

(actually ML, shhhh…)
- (unsupervised learning)
- (supervised learning)
- (reinforcement learning)

(actually ML, shhhh…)
2012; h/t michael littman

learnings

bit.ly/AlexCTM
recommendation as inference

Chong & Blei, SIGKDD 2011
CTM generative model:
r: clicks
w: words
u: user-topic association
v: article-topic association

Chong & Blei. SIGKDD 2011
related: learning phenotypes from EHR

UPDATE COPYReaderscope
In the course of our global expansion, we
realized we needed to have much more
sophisticated, real-time insight into what’s
happening across our site.  
Who is reading what? And where?

LOCATIONS FAQAUDIENCE SEGMENTSTOPICS
FAQ or Intro Information
Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor incididunt
ut quislabore et dolore magna aliqua. enim ad
minim veniam, quis quis aliqua ullamconostrud
exercitation ullamco laboris nisi ut aliquip ex ea
commodo consequat.
Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat
nulla pariatur. Excepteur sint occaecat non
proident, sunt in culpa qui officia deserunt
mollit anim id est laborum. aliquip ex ea
commodo consequat.
nulla pariatur. Excepteur sint occaecat
cupidatat non proident, sunt in culpa qui
officia deserunt mollit anim id est laborum.
aliquip ex ea commodo consequat.aliquip ex
Privacy Policy Terms of Service © The New York Times CompanyNYTimes.com Send Us Feedback
Searchex. High-tech Lifestyle, Parents, Media - Comedy Films
Audience Segment
Search by
nytreaderscope
Illustration by Clara Nguyen
commodo consequat.
commodo consequat.
Audience Segment
Search by
nytreaderscope
f
f
f
Tool: Readerscope
AUDIENCE INSIGHTS ENGINE

commodo consequat.
commodo consequat.
Audience Segment
Search by
nytreaderscope
commodo consequat.
commodo consequat.
Audience Segment
Search by
nytreaderscope
f
Tool: Readerscope

commodo consequat.
commodo consequat.
Audience Segment
Search by
nytreaderscope
C-Suite
C-Suite, Executives and BDMs - Entertainment
C-Suite, Executives and BDMs - Media
C-Suite|
C-Suite
C-Suite, Executives and BDMs - Entertainment
C-Suite, Executives and BDMs - Media
commodo consequat.
commodo consequat.
Audience Segment
Search by
nytreaderscope
f
Tool: Readerscope

predictive modeling, e.g.,
“the funnel”

interpretable predictive modeling
supercoolstuff

interpretable predictive modeling
supercoolstuff
Middendorf, Kundaje, Shah, Freund, CW, Leslie
arxiv.org/abs/q-bio/0701021

work w/Daeil Kim & Hiroko Tabuchi

work w/Daeil Kim & Hiroko Tabuchi
driving question: which records should she investigate?

We conducted
user research
to gather
millions of
observations
about how
different
articles made
people feel.
DATA COLLECTION
When reading this article, did you feel…
Anger Sadness Happiness Despair
Hurt No Emotion Jealousy Frustration
Anxiety Hope Hate Interest
Guilt Contentment Contempt Love
Compassion Shame Amusement Stress
Irritation Fear Boredom Surprise
Confusion Disgust Irony Pride
Disappointment

*units based on 100th percentile

Adventur
ous
98
Interest
42
Happiness
96
Self 
Confident
39
Love
97
*units based on 100th percentile
Hate
Inspired
100
Amused
100
Sadness
27

Sources: Google DFP, NYT Ad Performance Data, Sizmek
April May June July August September October November December
Perspective Targeting Impression Volume By Month
Throughout the year, NYT
began running more and more
perspective targeting
campaigns every month.
And performance kept
breaking boundaries and
setting new benchmarks for
success.
A Record First Year

Perspective Targeting
outperformed brands’ own
first-party segments,
demonstrating that we’ve
managed to find a new way to
deliver meaning to these
audiences.

learnings
…two examples

w/“audience development” team

leverage methods which are predictive yet performant
w/“audience development” team

driving question: which content
should we promote, where and when?

NB: data informed, not data-driven

learnings
… recommendation as prescription

2018: algos for *highly editorially curated* content pools
- smarter living
- midterms
- editors picks

2019: all of the above, plus:
- For You Tab
- stay tuned…

slow: Randomized controlled trial

old (1933) idea: do the best you can

Lihong Li (YHOO->MSFT->GOOG), 2011
thompson sampling & “bandits”

Related/extensions: lessons from statistical physics:
- Variational Methods for complex algebraic models
- Urteaga & W. arXiv:1709.0316 / AISTATS 2018
- Monte Carlo methods
- nonparametric mixture models: Urteaga & W. arXiv:
1808.02932 [stat.ML] via MC)  
- particle filtering for arbitrary computational
models: Urteaga & W. arXiv:1808.02933

approximate variational methods for bandits
Urteaga & W. arXiv:1709.03163

cf. modelingsocialdata.org
monte carlo filtering w/drift? -> ‘genetic’ bandit

dynamic environments
Urteaga & W. arXiv:1808.02932

common requirements in
data science:

common requirements in
data science:
1. people
2. ideas
3. things
cf. John Boyd, USAF

monica rogati, Aug 1 2017 hackernoon.com
things: de>da>ds/ml/ai

Reporting
Learning
Test
Optimizing
Exploredescriptive:
predictive:
prescriptive:

Reporting
Learning
Test
Optimizing
Exploredescriptive:
predictive:
prescriptive:
ML primitives: learning, scoring, testing;
- speed?
- scale?
- cost?

watch this space: NYT+AI
physics
math/fin p chem app math
cog sciEE

people.. so far (we’re hiring!!!!)
astrophys math/fin
pure mathapp math
cog sciEE
biophysseismology neuro
physics

chris.wiggins@hackNY.org
@chrishwiggins

chris.wiggins@hackNY.org
@chrishwiggins
also: we’re hiring!

Data Science at The New York Times: what industry can learn from us; what we can learn from industry

Recommended

Recommended

More Related Content

Similar to Data Science at The New York Times: what industry can learn from us; what we can learn from industry

Similar to Data Science at The New York Times: what industry can learn from us; what we can learn from industry (10)

More from chris wiggins

More from chris wiggins (20)

Recently uploaded

Recently uploaded (20)

Data Science at The New York Times: what industry can learn from us; what we can learn from industry