How Netflix Manages $18B of Content Spend With Brian Orth and David Johnson | Current 2022 Netflix spends an ever increasing amount on content to bring joy to our members – over 18 billion dollars in 2022! As a business, how does Netflix ensure that our forecasted spend is accurate? How do we enable systems and business processes to be able to move in a highly aligned, loosely coupled way that is so critical to the Netflix Culture? Come learn how Netflix Content Controllership Engineering uses Kafka to provide a highly available, ordered stream of data that allows our fleet of financial services to have a measurable increase in forecasting model accuracy by using data from Actual Spend. All while taking advantage of Netflix Engineering investment into our Studio API. We’ll go into the details of building observability into our event-driven architecture, talk through event data enrichment, and explain how we are building scalability and flexibility into our design.

1. David Johnson
Brian Orth

2. 2
Brian Orth
Eng @ Netflix
David Johnson
Eng @ Netflix
Speakers

3. 3
Q: How many countries did
Stranger Things Season 4
become #1 in the
week after launch?
Break The Ice

4. 4
Building largest Studio in the World
Roughly $18b in Content Spending
Netflix Animation Growing Rapidly
Expanding to Netflix Games!
Netflix Studio Scaling

5. 5
Launch
Production
Creative
Forecast Program Deals Post Production
Accounting
& Reporting
Content Finance Infrastructure
How Does Netflix Make Content?

6. 6
Our Studio World
● Scaled from 1 production to 1000’s
● Vertically Built Solutions
● Manual Offline Reviews
● Disconnected Finance

7. 7
● Break the problem down
● Share data of high quality
● Largest Business Impact (hours, $$$, risk)
Forecast
Accounting
& Reporting
Solutions

8. 8
● Netflix’s Paved Road
● Highly Aligned Loosely Coupled
● Freedom AND Responsibility
● Informed Captains for Decisions
Designing As A Netflix Engineer

9. 9
● Operate what you build
● Design for support and
maintainability
● Easy retries and backfill
● Observability
Strategic Thinking

10. 10
Request Driven Eventing
Decoupling
Traceability
Consistency
Uniform Reconciliation
Channels of
communication
Producers * Consumers Producers + Consumers
Trade Offs

11. 11
● GraphQL Schema
○ Federated Graph
○ JSON message type
○ gzip’ed
● Complete dataset per title
● Most current version
// Payload Event
{
"eventType": "UPDATE",
"eventId": "6edd1a0d-ec6b-413d-9d00-3b9e6cd05771",
"application": "accountingService",
"metadata": {
"dealId": 1,
"movieId": 1234,
"payload": <compressed> (Gzip + Base64 encoded)
},
"eventSource": "accountingService.enricher",
"timestamp":1664333644674
}
Data Modeling

12. 12
Application Database
Enrichment
Data Integrity
Checks
Iceberg
Forecasting
Accounting
System
Triggers Export
Of Raw Events
Event Routing
Service
Event Producer
Future Consumer

13. 13
● Flexible design to enable accounting workflows
● Traceability
● Lightweight
● Single Writer
Accounting
System
Triggers Export Of
Raw Events
Event Routing Service
Event Routing Service

14. 14
● Horizontally scalable
● Parallelizable
● GraphQL source of truth
Enrichment
Raw
Events
Enriched
Events
Enrichment

15. 15
● Decoupled
● Scalable
● Idempotent
Enrichment Data Integrity Checks
Iceberg
Forecasting
Consumer
Future Consumer

16. 16
● Retries
● Backfill/Replaying Events
● At least once semantics
Recovery & Operability

17. 17
Enrichment
Data Integrity
Checks
Future Consumer
Forecasting
Accounting
System
Triggers Export
Of Raw Events
Event Routing
Service
Case 1: Incomplete Export
Application Database
Fault Tolerance

18. 18
Enrichment
Data Integrity
Checks
Future Consumer
Forecasting
Accounting
System
Triggers Export
Of Raw Events
Event Routing
Service
Case 2: Fails to Publish
Application Database
Fault Tolerance

19. 19
Enrichment
Data Integrity
Checks
Future Consumer
Forecasting
Accounting
System
Triggers Export
Of Raw Events
Event Routing
Service
Case 3: DB Down
Application Database
Fault Tolerance

20. 20
Enrichment
Data Integrity
Checks
Future Consumer
Forecasting
Accounting
System
Triggers Export
Of Raw Events
Event Routing
Service
Case 4: Enrichment Service Down
Application Database
Fault Tolerance

21. 21
Enrichment
Data Integrity
Checks
Future Consumer
Forecasting
Accounting
System
Triggers Export
Of Raw Events
Event Routing
Service
Case 5: Enrichment Fails
Application Database
Fault Tolerance

22. 22
Enrichment
Data Integrity
Checks
Future Consumer
Forecasting
Accounting
System
Triggers Export
Of Raw Events
Event Routing
Service
Case 6: DB Fails (Regardless of Enrichment Status)
Application Database
Fault Tolerance

23. 23
● Alerting – report when something goes awry
● Discovery – determine what isn’t working
● Diagnosis – debug why something isn’t working
Observability

24. 24
Domain Challenges
● Translating data model between
producers and consumers
● Adding supplementary data
Technical Challenges
● Ordering / idempotence
● Failure cases
● At least once
Challenges

25. 25
● Able to answer business questions in near real-time
● Automated user workflows
● Consumer architecture decoupled
Results

26. 26
● Event Trigger from Application
○ Future Change Data Capture
● In the future Kafka can be source of truth
with log compaction
● Additional Consumers without needing to
change the architecture
● Streaming as the backbone of the Studio
Finance Connected Data
Coming Soon

27. 27
Thank you!
Contact us at
borth@netflix.com
davidjohnson@netflix.com
We Are Hiring!