http://www.oreilly.com/pub/e/3764 Keystone processes over 700 billion events per day (1 peta byte) with at-least-once processing semantics in the cloud. Monal Daxini details how they used Kafka, Samza, Docker, and Linux at scale to implement a multi-tenant pipeline in AWS cloud within a year. He'll also share plans on offering a Stream Processing as a Service for all of Netflix use.

1. A NETFLIX ORIGINAL SERVICE
Netflix Keystone - Cloud scale event processing pipeline
[ O’reilly - Turning big data into knowledge ]
Monal Daxini

2. ● Season 1
○ Keystone pipeline - Why? How? What?
● Season 2
○ Trailer
What to expect?

3. Monal Daxini
I lead the Stream Processing effort in the
Real Time Data Infrastructure team @ Netflix
@monaldax #Netflix #Keystone
Who?

4. Netflix Is a Data Driven Company
Content
Product
Marketing
Finance
Business Development
Talent
Infrastructure
←CultureofAnalytics→

5. Netflix Service
Truly Global Internet TV Network

6. Over 80 Million Members
190 Countries
1000+ devices

7. 125,000,000 hours/day
14,269 years worth / day
37% of Internet traffic at peak

8. Over
1,000,000,000,000
That’s a huge number!
Keystone Events Processed Every Day

9. Events Trend
1/2014 80B / day 1/2015 300B / day 1/2016 1T / day

10. Keystone Scale
Daily Averages
● 700B unique events ingested
● 1T events processed
● 1.5PB / day
● 4K / event
Peak
● 1T unique events ingested/day
● 12.5M / sec
● 35GB / sec
● 10MB / message

11. ● 99.99% / Four 9s
● Initial numbers with VPC
○ 99.9999% / Six 9s
Keystone Availability

12. Keystone Pipeline Evolution

13. In the beginning… Chukwa
EMR
Event
Producers

14. Q4 2014 - Chukwa / Suro
Event
Producer
Druid
Stream
Consumers
EMR
Consumer
Kafka
Suro Router
Event
Producer
Suro
Kafka
Suro
Proxy

15. ● Support at-least-once processing
● Scale, Multi-tenancy, Ease of Operations
● Enable future value adds - Stream Processing As a Service
● Replace dormant open source software - Chukwa
Why a new pipeline?

16. Goal - Migrate 1.3 PB of event data to a new Pipeline
in flight, while not losing more that 0.1% of them

17. Q4 2015 - Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Consumer
Kafka
Control Plane
Event
Producer
KSProxy

18. Want to know more...
Netflix Tech Blog - Pipeline Evolution

19. Event flow
Keystone Pipeline As a Service

20. Keystone
Stream
Consumers
Samza
Routers
EMR
Fronting
Kafka
Clusters
Event
Producer
Consumer
Kafka
Control Plane

21. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

22. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

23. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

24. Self Service UI (Keystone Management)

25. Events & Producers

26. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

27. Event Payload is Immutable
At-least-once semantics*
* Once the event makes it to Kafka, there are disaster scenarios where this breaks.

28. Injected Event Metadata
● GUID
● Timestamp
● Host
● App
`

29. Custom Extensible Wire Protocol
● Backwards and forwards compatibility
● Support multiple serialization formats
○ JSON, AVRO, Protobuf in the works
● Additional metadata
● Efficient - 10 bytes overhead per message

30. Netflix Kafka Producer
● Configurable - topic to Kafka clusters routing
● Sticky partitioner
● Prefer event drop than disrupt producer app
● Best effort delivery, ack = 1
● Buffer size tuning based on traffic

31. Kafka Clusters

32. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

33. Kafka (prod) Footprint
Fronting
Kafka
Front Standby Kafka Consumer Kafka
Number of Clusters 24 24 8
Number of Instances 3000+ 72 1000+
Retention Period 8 to 24 hrs 1 hr 2 to 4 hrs

34. ● Independent zookeeper cluster per Kafka cluster
● 5 nodes per ensemble - 160 zookeeper nodes
● 3 ASGs per cluster, 1 ASG per zone
Kafka (prod) Footprint

35. ● Pioneer Tax
● Started with 0.7, went live with 0.8.2
● Done moving to 0.9 & VPC
● Work closely with Confluent to get patches through
○ OSS contributions
Kafka in the Cloud

36. ● No dynamic topic creation
● Two copies
● Rack / Zone aware partition assignment
● Per Cluster Stay under 10k partitions & 200 brokers
● Leave approx. 40% free disk space on each broker
Fronting Kafka Topics

37. Want to know more...
Netflix Tech Blog - Kafka in Keystone Pipeline

38. Routing Service

39. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

40. Samza Job Deployment
● Multiple Samza jobs for one Kafka source topic
● Each job processes messages for one sink
● Each job processes partitions only from one topic
● One checkpoint topic per Kafka source topic and
multiple samza jobs
● Job starts with Immutable Config

41. ● Over 13,000+ docker containers (samza jobs)
● 1,300+ AWS C3-4XL instances
Routing Service Footprint
S3 ElasticSearch Consumer Kafka
Number of containers 7000+ 1500+ 4500+

42. Routing Latency
Fronting Kafka to Sinks
S3 ElasticSearch Consumer Kafka
1 sec 13 sec 800 ms

43. Routing Infrastructure
+
Checkpointing
Cluster
+
0.9.1
Go

44. Router Job Manager
(Control Plane)
EC2 Instances
Zookeeper
(Instance Id assignment)
Job
Job
Job
ksnode
Checkpointing Cluster
ASG

45. Custom Go
Executor
./runJob
Logs
Snapshots
Attach Volumes
./runJob
./runJob
Reconcile Loop - 1 min
Health Check
What’s running in ksnode?
Zookeeper
(Instance Id assignment)

46. Logs ZFS Volume
Snapshots
Custom Go
Executor
.
/runJo
b
.
/runJo
b
.
/runJo
b
Go Tools Server
Client
Tools
Stream Logs
Browse through
rotated logs by date
Ksnode Tooling

47. Yes! You inferred right!
No Mesos & No Yarn

48. Samza Tweaks to ver 0.9.1
● Using ThreadJobFactory - Simplifies deployment and reduces overhead
● SAMZA-41 - range based static partition range assignment
● SAMZA-775- size based Prefetch buffer
○ Default was count based (OOM), and not bytes based
○ Set per topic per job to 60 * peak bytes / sec over the past week.

49. Samza Tweaks to ver 0.9.1
● Backported from 0.10
○ SAMZA-655 - environment variable configuration rewriter
■ Pass config from RDS to executor to Docker to Samza Job
○ SAMZA-540 - expose latency related metrics in OffsetManager
■ checkpointed offset gauge

50. More Info - Monal’s Samza Meetup Slides (10/2015)
Netflix Samza ver 0.9.1 Contributions

51. Metrics & Monitoring

52. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Consumer
Kafka
Control Plane
Event
Producer
KSProxy

53. Customer Facing per topic end-to-end dashboard

54. Birth of Kafka Kong

55. A True Story
Keystone went live 10.27.2015
2 days later...

56. A True Story
● 80% Loss over 6 hour period
● Large Kafka clusters were impacts, smaller ones were fine

57. ● At times things go wrong, and there’s no turning back
● Reduce complexity
● Minimize blast radius
● Find a way to start over fresh
Lessons Learned

58. Samza Router
Fronting
Kafka
Event
Producer
X
Failover
Stand-In
Kafka

59. ● Cold standby Kafka cluster with 3 instances and different instance type
● Different ZooKeeper cluster with no state
● When failover occur
○ Scaling up cluster
○ Creating topic
○ Creating new routing jobs for failover cluster
○ Switch producer traffic!
Failover

60. Kafka Kong
At least once
a week

61. Fronting Kafka Failover
Self Service Tool

62. Fronting Kafka Failover
Fully
Automated
● Time is the essence - failover as fast as 5 minutes

63. Culture
What does it have to do with
building a pipeline?

64. "It may well be the most important document
ever to come out of the Valley." 1
Sheryl Sandberg
COO, Facebook
1 Business Insider, 2013

65. Netflix Culture Deck
Netflix Culture
Freedom & Responsibility

66. Open source and community participation is
an integral part of our strategy and culture

67. Not DevOps, but move towards NoOps
You build it! You run it!

68. My team has
● No dedicated product or project managers
● No separate devops or operations team
We build and run what you saw today!

69. ● This does not mean we are constantly overworked
○ we make wise and simple choices and
○ lean towards automation & self-healing systems
We build and run what you saw today!

70. We built a pipeline in a year with
A very small team,
Relevant new technology,
Contributed back to OSS, and
Processed over 1 Trillion messages / day
Culture Impact

71. Season 2
Trailer

72. Create DuploⓇ
Blocks:
Let reusability drive new value
Our Philosophy

73. Evolution
Keystone
Stream Processing
(SPaaS)
Keystone
Management
Keystone
Messaging
Schema Support

74. Simple and intuitive interface to
manage all Keystone services
Keystone Management

75. Keystone Stream Processing
Stream Processing As a Service (SPaaS)
Multi-tenant polyglot support for stream processing engines
BETA

76. Big Data Systems - streaming
Data Pipeline & Stream processing - Keystone - Samza / Flink (poc)
Playback & edge Operations insight - Mantis
Stream Processing - Spark Streaming
* Metrics & monitoring - Atlas *

77. SPaaS Architecture
SPaaS
Manager
Container
Runtime
Beam API
or
Framework Specific API
[ Dockerized Job ]
1. Create 2. Submit 3. Launch
Runner
Flink / Spark /
Mantis
Running
Job
1. Submit Job
DSL (SQL)
Job Dashboard
BETA

78. ● Starting out Proof Of Concept with Apache Flink
● Exploring Apache Beam
SPaaS - init( )
BETA

79. Why Apache Beam?
○ Portable API layer for building sophisticated data processing
applications
○ Unified APi for processing bounded and unbounded data sources
○ Google lineage - Dataflow model, and reflects Google’s current work
SPaaS - “Beam Me Up, Scotty ! "

80. Why Flink?
○ Flink implements the dataflow model
○ Correctness of results and powerful features for reasoning about time
○ Checkpoints, exactly-once processing
○ Event time, processing time, watermarks, triggers, aligned windows
(fixed, sliding), unaligned windows (dynamic or session windows)
○ Flink’s core is a streaming engine
SPaaS - Flink

81. More brain food...
Netflix OSS
Samza Meetup Presentation
Netflix Tech Blog
Spark Summit 2015 Talk

82. Thanks!
Q & A
You can find me at
@monaldax
mdaxini@netflix.com

83. ● Special thanks to all the people who made and released these awesome
resources for free:
○ Photographs by Unsplash
Credits