Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
confluent, profile picture
Uploaded byconfluent
3,096 views

Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Every Piece of Content Ever Published in Kafka

The document discusses how The New York Times stores all published content in Apache Kafka. It describes how the Times previously used an API-based architecture with different data schemas for each service, which led to inconsistencies. The new approach establishes a single source of truth (Kafka) with a normalized schema. All content is stored as a log in Kafka, then denormalized logs and special-purpose logs are replicated from the main log. This allows content to be efficiently accessed and ensures consistency across services. The implementation uses Google Cloud infrastructure and passive replication across regions. Key lessons are that log-based architectures are new, managed Kafka would help, and it will take time to transition all services.

Software◦
The Source of Truth 2017-05-08 The New York Times Why The New York Times Stores Every Piece of Content Ever Published in Kafka
Boerge Svingen Director of Engineering at the New York Times, working on backend systems.
Topic: How is published content made available to the front-end applications?
CMS CMS Archives Web iOS Android Etc. Producersofcontent Consumersofcontent Etc. Etc. Etc.
Agenda 1. A little history 2. How things used to work 3. The source of truth 4. Implementation 5. Lessons so far
Agenda 1. A little history 2. How things used to work 3. The source of truth 4. Implementation 5. Lessons so far
Source: http://www.nytimes.com/1865/04/15/news/president-lincoln- shot-assassin-deed-done-ford-s-theatre-last-night-act.html
20 years on the web
The New York Times Company Archives
The New York Times Company Archives
The New York Times Company Archives
The New York Times Company Archives
The New York Times Company Archives
The New York Times Company Archives
The New York Times Company Archives
Agenda 1. A little history 2. How things used to work 3. The source of truth 4. Implementation 5. Lessons so far
Producersofcontent Consumersofcontent Personal- ization CMS CMS Archives Web iOS Android Etc. Search Etc. Etc. Etc. Etc. Etc. Etc.
A rather typical API-based architecture.
Disadvantages with this approach … The consumers have to know about all the producers of content.
Disadvantages with this approach … Every API tends to be different.
Disadvantages with this approach … Every API tends to return data with a different (no) schema.
Disadvantages with this approach … We have no efficient way of reading old content in bulk, so it’s hard to replace service stores.
Disadvantages with this approach … Most services have to manage permanent state.
Disadvantages with this approach … It is difficult to change the (non-existent) schema, leading to inconsistencies and duplication.
Disadvantages with this approach … We get monoliths that try to be everything for everyone.
Disadvantages with this approach … It’s hard to develop new products and change current ones.
Agenda 1. A little history 2. How things used to work 3. The source of truth 3. Implementation 4. Lessons so far
The Publishing Pipeline
CMS CMS Archives Web iOS Android Etc. Producersofcontent Consumersofcontent Kafka Gateway Search Personal- ization Collections Etc. Etc. Etc. GraphQLAPI Etc. Etc. Etc.
We have a schema.
The schema. Uses proto3.
The schema. Is normalized.
Article 1 Dateline 1 Credit 1 Section 1 Image 2 Image 1 Credit 2 Article 2 Section 2 Image 3
The schema. The Gateway validates all assets before they go on the log.
The schema. All assets are identified by a URI: nyt://article/186faf12-24a0-4dda-b737-018cee0b81cd
The schema. Custom linter to check for forwards and backward compatibility.
The schema. GraphQL schema is automatically generated from the protobuf schema.
Monolog.
The Monolog Single partition, totally ordered, infinite retention.
The Monolog The Source of Truth for published content.
The Monolog Contains everything published since 1851.
Article 1 Dateline 1 Credit 1 Section 1 Image 2 Image 1 Credit 2 Article 2 Section 2 Image 3
Article 1 Dateline 1 Credit 1Section 1 Image 2 Image 1Credit 2 Article 2 Section 2 Image 3 Topological sort
Section1 Dateline1 Credit1 Credit2 Image1 Image2 Image3 Section2 Article2 Article1 Image2,version2 Credit2,version2
Denormalized log.
The denormalized log Replicated from the monolog.
The denormalized log Updates the full asset every time a dependency is updated.
The denormalized log Makes it easier for consumers that need all the dependencies.
The denormalized log Partitioned by asset ID.
Article1 Dateline1 Credit1 Section1 Image2 Image1 Credit2 Article2 Section2 Image3 Dateline1 Image1 Credit2 Article1 Dateline1 Credit1 Section1 Image2,version2 Image1 Credit2
Special-purpose logs
Special-purpose logs All replicated from the monolog.
Agenda 1. A little history 2. How things used to work 3. The source of truth 4. Implementation 5. Lessons so far
Runs on Google Cloud.
Producersofcontent Consumersofcontent Kafka broker Kafka broker Kafka broker Kafka broker Kafka broker ZooKeeper ZooKeeper ZooKeeper ZooKeeper ZooKeeper Gateway Gateway Gateway Gateway ReplicatorsReplicatorsReplicatorsReplicators GKE (Kubernetes) GKE (Kubernetes) Google Compute Engine gRPC over Cloud Endpoint Kafka Consumer over SSL
Passive replication.
Producersofcontent Consumersofcontent us-east us-central us-west GraphQLAPI Gateway Etc. Etc.Kafka consumers Gateway Etc. Etc.Kafka consumers Gateway Etc. Etc.Kafka consumers Producersofcontent Consumersofcontent us-east us-central us-west GraphQLAPI Gateway Etc. Etc.Kafka consumers Gateway Etc. Etc.Kafka consumers Gateway Etc. Etc.Kafka consumers
Agenda 1. A little history 2. How things used to work 3. The source of truth 4. Implementation 5. Lessons so far
Managed Kafka would be very nice.
(Assuming it would run on Google Cloud.)
Log-based architectures are still very new.
Google PubSub/SNS/SQS/Kinesis are not replacements for Kafka.
It’s still early days.
It will take us a while to move all services over to the new architecture.
Questions?