The document discusses Netflix's approach to handling big data problems. It summarizes Netflix's data pipeline system called Keystone that was built in a year to replace a legacy system. Keystone ingests over 1 trillion events per day and processes them using technologies like Kafka, Samza and Spark Streaming. The document emphasizes Netflix's culture of freedom and responsibility and how it helped the small team replace the legacy system without disruption while achieving massive scale.

1. The way
to deal with Big data problems
Monal Daxini
March 2016

2. Monal Daxini
Real Time Data Infrastructure
Senior Software Engineer, Netflix
https://www.linkedin.com/in/monaldaxini
@monaldax
#Netflix #Keystone

3. We help Produce,
Store,
Process,
Move
Events @ scale

4. Tell me more...
● Big Data Ecosystem @ Netflix
● How we built a scalable event pipeline - Keystone - in a year
○ Replaced legacy system without service disruption with a
small team.
● Netflix Culture
○ Relevant tenets tagged on the slides

5. Global Launch - Jan 6, 2016

6. Over 75M Members
190 Countries
125M hours/day → 11B hours / quarter
14,269 years / day → 1,255,707 years / quarter
1000+ devices
37% of Internet traffic at peak

7. Netflix Is a Data Driven Company
Content
Product
Marketing
Finance
Business Development
Talent
Infrastructure
←CultureofAnalytics→

8. Data @ Netflix
Data at Rest (batch)
Data in Motion (streaming)

9. Big Data Systems - batch
Ingestion / Kafka -> Ursula, Aegisthus
Storage / S3, Teradata, Redshift, Druid
Processing / Pig, Hive, Presto, Spark
Reporting / Microstrategy, Tableau, Sting
Scheduling / UC4
Interface / Big Data Portal, Kragle (API)
Opensource&
CommunityDriven

10. Big Data Systems - batch

11. Scale - batch
AWS S3 (instead of HDFS)
40 PB (S3) Compressed
Of which 13 PB events data

12. Big Data Systems - streaming
Data Pipeline - Keystone
Playback & edge operational insight - Mantis
Stream Processing* - Spark Streaming
Metrics & monitoring - Atlas
LooselyCoupled
HighlyAligned
Opensource&
CommunityDriven

13. What does culture have to do with big data?

14. Netflix Culture Deck
Netflix Culture
Freedom & Responsibility

15. "It may well be the most important document
ever to come out of the Valley." 1
Sheryl Sandberg
COO, Facebook
1 Business Insider, 2013

16. A NETFLIX ORIGINAL SERVICE
How we built an internal facing 1 trillion / day stream processing
cloud platform in a year, and how culture played a pivotal role
Freedom
&
Responsibility

17. Years ago...

18. In the Old Days ...
EMR
Event
Producers

19. Chukwa/Suro + Real-Time Branch

20. About a year ago ...

21. Chukwa / Suro + Real-Time Branch
Event
Producer
Druid
Stream
Consumers
EMR
Consumer
Kafka
Suro Router
Event
Producer
Suro
Kafka
Suro
Proxy

22. Support at-least-once processing
Scale, Ease of Operations
Replace dormant open source software - Chukwa
Enable future value adds - Stream Processing As a Service
Seamless transition to the new platform
ContextNotControl

23. Migrate Events to a new Pipeline In flight,
while not losing more that 0.1% of them
ContextNotControl
HighlyAligned
LooselyCoupled

24. Jan 2016

25. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Consumer
Kafka
Control Plane
Event
Producer
KSProxy

26. 1 trillion events ingested per day during holiday season
1+ trillion events processed every day
350 billion a year ago 600+ billion events ingested per day
Keystone - Scale - Streaming

27. 11 million events (24 GB per second) peak
Upto 10MB payload / Avg 4K
1.3 PB / day
Keystone - Scale - Streaming

28. Events & Producers

29. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

30. Event Payload is Immutable
At-least-once semantics*
* Once the event makes it to Kafka, there are disaster scenarios where this breaks.

31. Injected Event Metadata
● GUID
● Timestamp
● Host
● App

32. Keystone Extensible Wire Protocol
● Backwards and forwards compatibility
● Supports JSON, AVRO on the horizon
● Invisible to source & sinks
● Efficient - 10 bytes overhead per message
○ because message size - hundreds of bytes to 10MB

33. Netflix Kafka Producer
● Best effort delivery - ack = 1
● Prefer drop event than disrupting producer app
● Resume event production after Kafka cluster restore
● Integration with Netflix Ecosystem
● Configurable topic to Kafka clusters route

34. Fronting Kafka Clusters

35. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

36. ● Pioneer Tax
● Started with 0.7
● In prod with 0.8.2
● Move to 0.9 & VPC in progress
Kafka in the Cloud

37. Based on topics assigned
● Normal-priority (majority)
● High-priority (streaming activities etc.)
Fronting Kafka Topic Classification

38. ● ≅3200 d2.xl brokers for regular, failover, & consumer
kafka clusters
● 125 Zookeeper nodes (25 ensembles)
○ Independent zookeeper cluster per Kafka cluster
● 24 island clusters, 8 per region
○ 3 ASGs per cluster, 1 ASG per zone
○ 24 warm standby 3 node failover clusters
Scale - Kafka (prod)

39. ● No dynamic topic creation
● Two copies
● Zone aware assignment of Topic partitions and replica
Fronting Kafka Topics

40. In a distributed system make sure you
understand limitations and failures,
even if you don’t know all the features.
- Monal Daxini

41. Because an unknown feature does not cause
failure, however an unknown failure mode
or limitation can bring down your system.
- Monal Daxini

42. In addition, we do
Kafka Kong once a
week

43. Fronting Kafka Failover
Self Service Tool
BlamelessCulture

44. Fronting Kafka Failover

45. Fronting Kafka Failover

46. Kafka Management UI (Beta)
Open sourcing on the road map
Opensource&
CommunityDriven

49. Kafka Auditor
Open sourcing on the road map
Opensource&
CommunityDriven

50. Kafka Auditor - One pre cluster
● Broker monitoring
● Consumer monitoring
● Heart-beat & Continuous message latency
● On-demand Broker performance testing
● Built as a service deployable on single or multiple instances

51. Kafka Cluster Size -Tips
● Per Cluster Stay under 10k partitions & 200 brokers
● Leave approx. 40% free disk space on each broker

52. ● Started with AWS zone aware partition assignments
● We have discovered and filed several bugs
○ Details - Upcoming in Netflix Tech blog
Kafka Contributions
Opensource&
CommunityDriven

53. Routing Service

54. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

55. Routing Infrastructure
+
Checkpointing
Cluster
+ 0.9.1Go
C language

56. Router Job Manager
(Control Plane)
EC2 Instances
Zookeeper
(Instance Id assignment)
Job
Job
Job
ksnode
Checkpointing Cluster
ASG

57. Custom Go
Executor
./runJob
Logs
Snapshots
Attach Volumes
./runJob
./runJob
Reconcile Loop - 1 min
Health Check
What’s running in ksnode?
Zookeeper
(Instance Id assignment)

58. Logs
ZFS Volume
Snapshots
Custom Go
Executor
.
/runJo
b
.
/runJo
b
.
/runJo
b
Go Tools Server
Client Tools
Stream Logs
Browse through
rotated logs by date
Ksnode Tooling

59. Yes! You inferred right!
No Mesos & No Yarn

60. Distributed Systems are Hard
Keep it Simple
Minimize Moving Parts

61. ● 13,000 docker containers (samza jobs)
○ 7,000 - S3 Sink
○ 4,500 - Consumer Kafka sink
○ 1,500 - Elasticsearch sink
● 1,300 AWS C3-4XL instances
Scale - Routing Service

62. More Info - Samza Meetup (10/2015)
Samza ver 0.9.1 Contributions
Opensource&
CommunityDriven

63. Target & Achieved <= 0.1% diff
bw Chukwa & Keystone pipeline,
over 2.6 PB of data / day
Chukwa & Keystone Pipeline Shadowing

64. Metrics & Monitoring

65. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Consumer
Kafka
Control Plane
Event
Producer
KSProxy

66. Customer Facing per topic end-to-end dashboard

67. Dev facing infrastructure end-to-end dashboard

68. Scaling Avenues

69. ● Exposed cost attribution per event producers & topic
○ E.g. one producer reduced throughput by 600%
● Automation - frees up additional resources
Scaling Up by Scaling Down
Oxymoron?

70. ● No dedicated product or project managers
● No separate devops or operational team
● Our team works this way, and most other infrastructure teams as well
● Netflix does have product and project managers, each team decides
independently
We build and run what you saw today!
YoubuildIt!
Yourunit!
HighPerformance

71. ● This does not mean we are constantly overworked
○ we make wise and simple choices and
○ lean towards automation & self-healing systems.
We build and run what you saw today!
YoubuildIt!
Yourunit!
HighPerformance

72. Not DevOps, but move towards NoOps
You build it! You run it!

73. ● High Performance culture
● Communication
● Honest & respectful feedback
● No culture of process adherence, but
○ Creativity & Self Discipline
○ Freedom and Responsibility

74. Open source and community participation is
an integral part of our strategy and culture

75. ● Collaborate with other internet scale tech companies
● Prevent lock-in of closed source products
● Need the flexibility to achieve scalability / functionality
Why use open source?

76. ● Help shape direction of projects
● Don’t want to fork and diverge
● Give back to the community
● Attract top talent
Why contribute back ?

77. ● Share our goodness
● Set industry standard
● Community can help evolve the tool
Why contribute our own tool ?

78. Looking into the future?

79. Streaming Processing As a Service
● multi-tenant polyglot support of streaming engines like
Spark Streaming, Mantis, Samza, and may be Flink
Future steps
Opensource&
CommunityDriven

80. Messaging As a Service
● Kafka & Others
● Spark Streaming, Mantis, Samza, and may be Flink.
Future steps
Opensource&
CommunityDriven

81. Data thruway
● Support for schemas - registry, discovery, validation.
Self Service Tooling
Future steps
Opensource&
CommunityDriven

82. More brain food...
Netflix OSS
Samza Meetup Presentation
Netflix Tech Blog
Spark Summit 2015 Talk