Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

The Source of Truth
2017-05-08
The New York Times
Why The New York Times
Stores Every Piece of Content
Ever Published
in K...

Boerge Svingen
Director of Engineering
at the New York Times,
working on backend systems.

Topic:
How is published content
made available to the front-end
applications?

CMS
CMS
Archives
Web
iOS
Android
Etc.
Producersofcontent
Consumersofcontent
Etc.
Etc.
Etc.

Agenda
1. A little history
2. How things used to work
3. The source of truth
4. Implementation
5. Lessons so far

Agenda
1. A little history
2. How things used to work
3. The source of truth
4. Implementation
5. Lessons so far

Source: http://www.nytimes.com/1865/04/15/news/president-lincoln-
shot-assassin-deed-done-ford-s-theatre-last-night-act.ht...

20 years on the web

The New York Times Company Archives

The New York Times Company Archives

The New York Times Company Archives

The New York Times Company Archives

The New York Times Company Archives

The New York Times Company Archives

The New York Times Company Archives

Agenda
1. A little history
2. How things used to work
3. The source of truth
4. Implementation
5. Lessons so far

Producersofcontent
Consumersofcontent
Personal-
ization
CMS
CMS
Archives
Web
iOS
Android
Etc.
Search
Etc.
Etc.
Etc.
Etc.
E...

A rather typical API-based
architecture.

Disadvantages with this approach …
The consumers have to know about all the
producers of content.

Disadvantages with this approach …
Every API tends to be different.

Disadvantages with this approach …
Every API tends to return data with a
different (no) schema.

Disadvantages with this approach …
We have no efficient way of reading old
content in bulk, so it’s hard to replace
servic...

Disadvantages with this approach …
Most services have to manage permanent
state.

Disadvantages with this approach …
It is difficult to change the (non-existent)
schema, leading to inconsistencies and
dup...

Disadvantages with this approach …
We get monoliths that try to be everything
for everyone.

Disadvantages with this approach …
It’s hard to develop new products and
change current ones.

Agenda
1. A little history
2. How things used to work
3. The source of truth
3. Implementation
4. Lessons so far

The Publishing Pipeline

CMS
CMS
Archives
Web
iOS
Android
Etc.
Producersofcontent
Consumersofcontent
Kafka
Gateway
Search
Personal-
ization
Collect...

We have a schema.

The schema.
Uses proto3.

The schema.
Is normalized.

Article 1
Dateline 1
Credit 1
Section 1
Image 2
Image 1
Credit 2
Article 2
Section 2
Image 3

The schema.
The Gateway validates all assets before
they go on the log.

The schema.
All assets are identified by a URI:
nyt://article/186faf12-24a0-4dda-b737-018cee0b81cd

The schema.
Custom linter to check for forwards and
backward compatibility.

The schema.
GraphQL schema is automatically
generated from the protobuf schema.

Monolog.

The Monolog
Single partition, totally ordered, infinite
retention.

The Monolog
The Source of Truth for published content.

The Monolog
Contains everything published since 1851.

Article 1
Dateline 1
Credit 1
Section 1
Image 2
Image 1
Credit 2
Article 2
Section 2
Image 3

Article 1
Dateline 1 Credit 1Section 1
Image 2
Image 1Credit 2
Article 2 Section 2 Image 3
Topological sort

Section1
Dateline1
Credit1
Credit2
Image1
Image2
Image3
Section2
Article2
Article1
Image2,version2
Credit2,version2

Denormalized log.

The denormalized log
Replicated from the monolog.

The denormalized log
Updates the full asset every time a
dependency is updated.

The denormalized log
Makes it easier for consumers that need all
the dependencies.

The denormalized log
Partitioned by asset ID.

Article1
Dateline1
Credit1
Section1
Image2
Image1
Credit2
Article2
Section2
Image3
Dateline1
Image1
Credit2
Article1
Datel...

Special-purpose logs

Special-purpose logs
All replicated from the monolog.

Agenda
1. A little history
2. How things used to work
3. The source of truth
4. Implementation
5. Lessons so far

Runs on Google Cloud.

Producersofcontent
Consumersofcontent
Kafka
broker
Kafka
broker
Kafka
broker
Kafka
broker
Kafka
broker
ZooKeeper ZooKeeper...

Passive
replication.

Producersofcontent
Consumersofcontent
us-east
us-central
us-west
GraphQLAPI
Gateway
Etc.
Etc.Kafka
consumers
Gateway
Etc.
...

Agenda
1. A little history
2. How things used to work
3. The source of truth
4. Implementation
5. Lessons so far

Managed Kafka
would be very nice.

(Assuming it would run on Google Cloud.)

Log-based architectures
are still very new.

Google PubSub/SNS/SQS/Kinesis are not
replacements for Kafka.

It’s still early days.

It will take us a while to move all services over
to the new architecture.

Questions?

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1044477 views
Storytelling For The Web: Integrate... by Chiara Aliotta 957527 views
Artificial Intelligence, Data and C... by OECD Directorate ... 884355 views
How to Leverage AI to Boost Employe... by SocialHRCamp 372649 views
2024 State of Marketing Report – by... by Marius Sescu 211589 views
Everything You Need To Know About C... by Expeed Software 230847 views

View on Slideshare

1 of 67 Ad

View on Slideshare

1 of 67 Ad