The document outlines the evolution of Netflix's real-time data infrastructure, focusing on the development of a collaborative ETL platform called Keystone. It emphasizes the importance of improving content production efficiency and enhancing customer experience through data-driven insights and streamlined processes. Key personnel are introduced, highlighting their roles and motivations in utilizing technology to drive better recommendations, analytics, and operational workflows.

1. Zhenzhong Xu
Real-Time Data Infrastructure @ Netflix
Evolving Keystone to
an Open Collaborative
Real-time ETL Platform

2. Evolving Keystone to
an Open Collaborative
Real-time ETL Platform
Zhenzhong Xu
Real-Time Data Infrastructure

3. Making the world smile
Netflix is to bring diverse stories around the world to our global members, to entertain everyone and make the world smile!

8. and she is not alone ...

9. Content
Production 101
Bring joy to our members:
● Best in class experience
Give the customer the freedom,
flexibility and best in class
experience to enjoy the
entertainment.

10. Content
Production 101
Bring joy to our members:
● Best in class experience
● Produce more exciting content
Build a technology driven
Studio to empower storytelling
and content production.

11. Zhenzhong Xu
Publish, Collect, Move & Compute event data
in near real time @ Cloud Scale
Stream
Consumers
Router
Batch System
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane
Self Service UI
Mantis

12. Keystone is ...
… a single self-contained PaaS
Event Processing
Pipeline

13. Keystone is ...
… a multi-tenants, self-serving tool

14. Keystone is ...
Stream
Processing
Service
Transport /
Messaging
Service
Producer API
Control Plane
Consumer
API
Self Service UI
… powered by a collection of building blocks

15. Putting together ...
Stream
Consumers
Router
Batch System
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane
Self Service UI
mantis

16. Elliot is a data scientist works
in the Data Science
Engineering organization. His
main motivation is to bring the
magic out of customer data and
improve customer experience.
Charlie is an application
developer who works in Studio
organization. His main
motivation is leverage
technology to improve the
content production business.
Introducing our super heros ...

17. Elliot’s work will result in
better data powered analytic
engine, drive better
recommendation and
personalization features for
Emily.
Charlie’ work will increase
content production workflow
efficiency, and ultimately help
Eleven enjoy higher quality
content.

18. Data engineers/scientists prefers simplicity and faster turnaround time to
be effective at their work.
Lots of common data engineering patterns are not generalized.
The A/B test we are doing today takes 28 days to
complete, we don’t have a way to detect early issues
so we can optimize our experiments!
Elliot

19. Recommendation & personalization

20. Which artwork
to show?

21. A/B testing

23. Real time
Reporting
Real time
Alerting
Faster training of
ML models
Resource
Efficiency
Why Stream Processing?

24. Anatomy of a stream processing job

26. Stream Processing connector ecosystem in Netflix
● Hive
● Iceberg
● Kafka
● Elasticsearch
● ...more coming
// Example in Java
DataStream<YourOutputDataType> dataStream = ...
// attach Iceberg sink to the DAG
getSinkBuilder()
.toIceberg("<sink_name>", dataStream)
.buildAndAddSink();
// Example in Scala
val srcStream = getSourceBuilder
.fromKafka("example-kafka-source")
.buildScala
.map(r => r.getPayload)
getSinkBuilder.toIcebergScala[util.Map[String,Object]](config,
"iceSink", srcStream)
.buildAndAddSink()

30. 3030
public class SpaasApplication extends SpaasBaseApplication {
@Override
public void constructJobDag(SpaasConfig config, StreamExecutionEnvironment env) {
// build a kafka source
SingleOutputStreamOperator<Record<Map<String, Object>>> sourceStream =
getSourceBuilder().fromKafka("kafkasource").build();
// dynamically pick the selected sink during deployment time
SinkFunction<Record<Map<String, Object>>> sink = getSinkBuilder().toSelector("dynamicsink")
.declareWith("noopsink", new NoopSink<Record<Map<String, Object>>>())
.or("kafkasink", getSinkBuilder().toKafka("kafkasink").build())
.build();
sourceStream.addSink(sink);
}
}
spaas.myTemplateNameSpace.source.names=dynamicsource
# "dynamicsource" is a dynamic selector that currently configured to pick kafka source
# the selector.selected configure can be override on SPaaS UI at runtime, and take effect after relaunch of the
job.
spaas.myTemplateNameSpace.source.dynamicsource.type=selector
spaas.myTemplateNameSpace.source.dynamicsource.selector.selected=kafka
spaas.myTemplateNameSpace.source.dynamicsource.selector.candidates=kafka,hive
Dynamic Source / Sink selector

33. 3333
@Override
public void constructJobDag(SpaasConfig config, StreamExecutionEnvironment env) {
ObjectMapper myObjectMapper = new ObjectMapper();
DeserializationSchema<Person> deserializer = new PersonDeserializer();
TypedSerializationSchema<Person, Record<Map<String, Object>>> serializer= new PersonSerializer();
SingleOutputStreamOperator<Record<Person>> sourceFunction =
getSourceBuilder()
.fromKafka("kafka")
.withOutputType(Person.class)
.withDeserializer(deserializer)
.build();
SinkFunction<Person> sinkFunction =
getSinkBuilder()
.toNull("null")
.withType(Person.class)
.withSerializer(serializer)
.build();
sourceFunction.map(Record::getPayload).addSink(sinkFunction);
}
}
Record Abstractions

34. … and more

35. A
B
C
A Map C
Filter A
Filter B
Filter C
Extractor
Pattern
Join
Pattern
Enrichment
Pattern
Data Engineering ETL patterns

36. Lots of applications are built to support studio production, but performing
consistent data synchronization is hard.
Data are spread in different space/time, making data search and discovery
hard.
I can’t believe it takes a team weeks of time to
find out how many scripts are written by female
writers!
Charlie

37. LaunchProductionCreativeForecast Program Deals Post Production Financial
Reporting

39. Heterogeneous Data
Synchronization
Operation
Reporting
Entity Search &
Discovery
Challenges

40. A case study: entity search

41. New event driven alternative ...

43. Challenge 1: Ordering Semantics

45. Me,
4 years old
My uncle, 2
years old

46. When forcing a global generation order...
Can Bob and Dave be logically the
same generation?

47. Revisit the ancestry
tree

48. The cone shape shows the
causal/partial ordering from
Dave’s frame of reference.

49. The light cone
representing the past,
present, and future ...
https://en.wikipedia.org/wiki/Light_cone

50. In a distributed system, it is
sometimes impossible to say
that one of two events
occurred first. The relation
“happened before” is
therefore only a partial
ordering of the events in the
system.

51. Figure referenced from wikipedia: https://en.wikipedia.org/wiki/Vector_clock

53. Figure on the right referenced from DataStax: https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlHowDataWritten.html
Multi-Master replication

56. Challenge 2: Processing Contracts

57. Message Contract

58. { // Infra layer (Chaski)
"magicByte": 0x01, // hex
"version": "0x12", // hex
"attributes": {
"id": "chaski-id-from-keystone",
"app": "example-app",
"host": "localhost",
"timestamp": 1234, // long
},
"payload": { // Platform layer (PlatformRecord)
"id": 1,
"operation_ts_utc_usec": 1234, // long
"extended_attributes": {
"netflix/data.platform/data.mesh/cassandra": { // Platform Developer-defined Avro record.
"schema_id": 1,
"payload": {
"encryption_key": "some-rsa-pub"
}
}
},
"operation": "UPDATE",
"payload": { // User layer. Represents state of the event after an update operation occurred upstream at the source.
"encrypted": false,
"format": "AVRO",
"schema_id": 1,
"payload": {
"row_id": 1,
"partition_id": 1,
"first_name": "net",
"last_name": "flix"
},
"remote_payload": null
},
"secondary_payload": null // User layer. Represents state of the event before the operation occurred upstream at the source. This
example is null to show that the source doesn't support pre-images.
}
}
Message
Contract

59. { // Infra layer (Chaski)
"magicByte": 0x01, // hex
"version": "0x12", // hex
"attributes": {
"id": "chaski-id-from-keystone",
"app": "example-app",
"host": "localhost",
"timestamp": 1234, // long
},
"payload": { // Platform layer (PlatformRecord)
"id": 1,
"operation_ts_utc_usec": 1234, // long
"extended_attributes": {
"netflix/data.platform/data.mesh/cassandra": { // Platform Developer-defined Avro record.
"schema_id": 1,
"payload": {
"encryption_key": "some-rsa-pub"
}
}
},
"operation": "UPDATE",
"payload": { // User layer. Represents state of the event after an update operation occurred upstream at the source.
"encrypted": false,
"format": "AVRO",
"schema_id": 1,
"payload": {
"row_id": 1,
"partition_id": 1,
"first_name": "net",
"last_name": "flix"
},
"remote_payload": null
},
"secondary_payload": null // User layer. Represents state of the event before the operation occurred upstream at the source. This
example is null to show that the source doesn't support pre-images.
}
}
Message
Contract

60. { // Infra layer (Chaski)
"magicByte": 0x01, // hex
"version": "0x12", // hex
"attributes": {
"id": "chaski-id-from-keystone",
"app": "example-app",
"host": "localhost",
"timestamp": 1234, // long
},
"payload": { // Platform layer (PlatformRecord)
"id": 1,
"operation_ts_utc_usec": 1234, // long
"extended_attributes": {
"netflix/data.platform/data.mesh/cassandra": { // Platform Developer-defined Avro record.
"schema_id": 1,
"payload": {
"encryption_key": "some-rsa-pub"
}
}
},
"operation": "UPDATE",
"payload": { // User layer. Represents state of the event after an update operation occurred upstream at the source.
"encrypted": false,
"format": "AVRO",
"schema_id": 1,
"payload": {
"row_id": 1,
"partition_id": 1,
"first_name": "net",
"last_name": "flix"
},
"remote_payload": null
},
"secondary_payload": null // User layer. Represents state of the event before the operation occurred upstream at the source. This
example is null to show that the source doesn't support pre-images.
}
}
Message
Contract

61. ● Processor Metadata
● Configurations Management
● Processor capabilities
● Operation responsibilities
Processor Contract

62. A simple use case: notify upon new deal!

63. Open Composable Processors

64. Bring it all together - rethink ETL ...

65. Keystone
Routers
Flink
Platform

66. Keystone
Routers
Flink
Platform
Don’t waste
time here!
High Value!
Our first stab - an Open,
Collaborative,
Composable, Configurable ETL
Platform...

69. Unleash creativity via an open collaborative data platform @ Netflix

70. THANKS