The document provides an overview of Keystone, Netflix's event processing pipeline. Some key points: - Keystone is a collection of microservices and components that form a single, self-contained logical service for processing over 500 billion events generated daily at Netflix. - It acts as a self-scaling, multi-tenant event processing pipeline that embraces continuous integration/continuous delivery to be self-healing and cloud failure tolerant. - The routing infrastructure uses Zookeeper for instance assignment and checkpoints to clusters stored in S3 for at-least-once delivery semantics under failure conditions. - The control plane handles container resource allocation, scheduling, and cluster orchestration and deployments. - Current

1. Keystone
Event Processing Pipeline
Zhenzhong Xu
on a Dockerized Microservices Architecture

2. Real-time Data Infrastructure – Netflix
Cloud Infrastructure - Microsoft
About Me

3. About Netflix
● 83M+ Subscribers
● 125M+ Streaming Hours / Day
● > 1/3 Peak NA Internet Traffic
● Thousands of Device Types
● Many Tens of Thousands of VMs
● 3 Active-Active Regions Across the World

4. Observe
Orient
Decide
Act CD

5. Observe
Orient
Decide
Act
Innovation
CD

6. Observe
Orient
Decide
Act
Innovation
Big Data
CD

7. Observe
Orient
Decide
Act
Innovation
Big Data
Culture
CD

8. Observe
Orient
Decide
Act
Innovation
Big Data
Culture
Cloud
CD

9. Microservices Ecosystem

10. Why a Event Processing Platform in Netflix?
● 500+ Billion events generated per day
● 1+T events processed per day
○ >1 PB
○ 4M – 16M / sec
○ 13GB - 43GB /sec
● Message Payload: 3 kb - 10mb

11. Data Driven Culture
● Realtime System Failure Detection
● A/B Testing
● Recommendation Algorithm
● Fraud Detection
● Distributed Tracing
● Log Quering

12. Paved Road in a Microservices Ecosystem
Microservices
produces events
Storage service,
and Batch/Stream
Processing
services
Event
Processing
Pipeline

13. Paved Road in a Microservices Ecosystem

14. Supports Batch & Streaming

15. Evolution of Netflix Keystone Pipeline

16. In the Old Days ...
EMR
Event
Producers

17. About a year ago
Event
Producer
Druid
Stream
Consumers
EMR
Consumer
Kafka
Suro Router
Event
Producer
Suro
Kafka
Suro
Proxy

18. Today
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Consumer
Kafka
Control
Plane
Event
Producer
KSProxy
Self Service
UI

19. Event flow
Keystone Pipeline As a Service

20. Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control
Plane
Self Service
UI

21. Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control
Plane
Self Service
UI

22. Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control
Plane
Self Service
UI

23. Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control
Plane
Self Service
UI

24. Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control
Plane
Self Service
UI

25. What exactly is Keystone?

26. Keystone is ...
… a collection of microservices & components
Stream
Processing
Service
Elastic
Pub/Sub
Queue
Producer
API
Control
Plane
Consumer
API
Self Service
UI

27. Keystone is ...
… a single self-contained logical service
Event
Processing
Pipeline

28. Keystone is ...
… an self-scaling, multi-tenancy service
that embraces CI/CD

29. Keystone is ...
… a self healing, cloud failure tolerant
service, guarantees at-least-once delivery
semantics

30. Let’s drill down ...
For the purposes of this talk, we’ll focus on...
Stream
Processing
Service
Elastic
Pub/Sub
Queue
Producer
SDK
Control
Plane
Consumer
SDK
Self Service
UI

31. Overview

32. Self
Service UI

33. Routing Infrastructure

34. Routing Infrastructure
EC2 Instances
Zookeeper
(Instance Id
assignment)
Job
Job
Job
ksnod
e
Checkpointing
Cluster
Server Group (Cluster)
Store logs in
S3

35. Routing Infrastructure
+
Checkpointing
Cluster
+
0.9.1Go
C language

36. Control
Plane
Custom Cluster Orchestration and Scheduling
Layer

37. Control
Plane
• Decides container resources
• Schedules container placements
• Orchestrates cluster deployments

38. Design Decisions?
Distributed System is all about trade-offs.

39. Container
● Process Isolation
● Fast Startup

40. Service Protocol
● Declarative
● Idempotent
● Reconciliation

41. State Management
● Stateless vs Stateful service
● Single source of truth

42. Scaling
● Self Scaling
● Partition boundary
● Idempotent operations
● Immutable server deployments

43. Delivery Semantics
● At-most-once
● At-least-once (best effort)
● Exactly-once

44. At-least-once under failure condition
● Checkpointing mechanism
● Optimize for writes
● Occasional reads

45. Multi-tenancy
● Isolation
● Heterogenous
● Cluster fragmentation

46. Failure Recovery
• Back pressure
• Network blip
• Container level failure
• Instance level failure
• Zone level failure
• Cluster level failure - Kafka-Kong
• Regional failure - Chaos-Kong

47. Stream Processing Engine
• Discovery integration
• Custom wire format integration
• Samza: Per partition serialized process loop
• Samza: Simple payload transformation
• Plugable abstraction

48. Current Scale - Routing Service
● 14,000 + docker containers
● 1,400 + EC2 C3-4XL instances
● 3 regions

49. Future Improvements
● Integrate with more sophisticated
orchestration/scheduling/cluster
management ecosystem
● Unlock value in real-time unbounded data
streams
● Data Discovery
● Data Silos

50. Questions?