Talk on Netflix Keystone by Peter Bakas at SF Data Engineering Meetup on 2/23/2016. Topics covered: - Architectural design and principles for Keystone - Technologies that Keystone is leveraging - Best practices http://www.meetup.com/SF-Data-Engineering/events/228293610/

1. A NETFLIX ORIGINAL SERVICE

2. Peter Bakas | @peter_bakas
@ Netflix : Cloud Platform Engineering - Real Time Data Infrastructure
@ Ooyala : Analytics, Discovery, Platform Engineering & Infrastructure
@ Yahoo : Display Advertising, Behavioral Targeting, Payments
@ PayPal : Site Engineering and Architecture
@ Play : Advisor to Startups (Data, Security, Containers)
Who is this guy?

3. common data pipeline to collect, transport, aggregate, process and visualize events
Why are we here?

4. ● Architectural design and principles for Keystone
● Technologies that Keystone is leveraging
● Best practices
What should I expect?

5. Let’s get down to business

6. Netflix is a logging company

7. that occasionally streams movies

8. 600+ billion events ingested per day
11 million events (24 GB per second) peak
Hundreds of event types
Over 1.3 Petabyte / day
Numbers Galore!

9. But wait, there’s more

10. 1+ trillion events processed every day
1 trillion events ingested per day during holiday season
Numbers Galore - Part Deux

11. How did we get here?

12. Chukwa

13. Chukwa/Suro + Real-Time Branch

14. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane
HTTP
PROXY

15. Kafka Primer
Kafka is a distributed, partitioned, replicated commit log service.

16. Kafka Terminology
● Producer
● Consumer
● Topic
● Partition
● Broker

18. Netflix Kafka Producer
● Best effort delivery
● Prefer msg drop than disrupting producer app
● Wraps Apache Kafka Producer
● Integration with Netflix ecosystem: Eureka, Atlas, etc.

19. Producer Impact
● Kafka outage does not disrupt existing instances from serving its purpose
● Kafka outage should never prevent new instances from starting up
● After kafka cluster restored, event producing should resume automatically

20. Prefer Drop than Block
● Drop when buffer is full
● Handle potential blocking of first meta data request
● ack=1 (vs 2)

21. Sticky Partitioner
● Batching is important to reduce CPU and network I/O on brokers
● Stick to one partition for a while when producing for non-keyed messages
● “linger.ms” works well with sticky partitioner

22. Producing events to Keystone
● Using Netflix Platform logging API
○ LogManager.logEvent(Annotatable): majority of the cases
○ KeyValueSeriazlier with ILog#log(String)
● REST endpoint that proxies Platform logging
○ ksproxy
○ Prana sidecar

23. Injected Event Metadata
● GUID
● Timestamp
● Host
● App

24. Keystone Extensible Wire Protocol
● Invisible to source & sinks
● Backwards and forwards compatibility
● Supports JSON. AVRO on the horizon
● Efficient - 10 bytes overhead per message
○ message size - hundreds of bytes to 10MB

25. Keystone Extensible Wire Protocol
● Packaged as a jar
● Why? Evolve Independently
○ event metadata & traceability metadata
○ event payload serialization

26. Max message size 10MB
● Keystone drops if > 10MB
○ Immutable event payload

27. Fronting Kafka Clusters

28. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

29. Fronting Kafka Clusters
● Normal-priority (majority)
● High-priority (streaming activities etc.)

30. Fronting Kafka Instances
● 3 ASGs per cluster, 1 ASG per zone
● 3000 d2.xl AWS instances across 3 regions for regular & failover traffic

31. Partition Assignment
● All replica assignments zone aware
○ Improved availability
○ Reduce cost of maintenance

32. Kafka Fault Tolerance
● Instance failure
○ With replication factor of N, guarantee no data loss with N-1 failures
○ With zone aware replica assignment, guarantee no data loss with multiple instance failures in the same
zone
● Sink failure
○ No data loss during retention period
● Replication is the key
○ Data loss can happen if leader dies while follower AND consumer cannot catch up
○ Usually indicated by UncleanLeaderElection metric

33. Kafka Auditor as a Service
● Broker monitoring
● Consumer monitoring
● Heart-beat & Continuous message latency
● On-demand Broker performance testing
● Built as a service deployable on single or multiple instances

34. Current Issues
● By using the d2-xl there is trade off between cost and performance
● Performance deteriorates with increase of partitions
● Replication lag during peak traffic

35. Routing Service

36. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

38. Broker

39. Routing Infrastructure
+
Checkpointing
Cluster
+ 0.9.1

40. Router
Job Manager
(Control Plane)
EC2 Instances
Zookeeper
(Instance Id assignment)
Job
Job
Job
ksnode
Checkpointing
Cluster
ASG
Reconcile every min.

41. Routing Layer
● Total of 13,000 containers on 1,300 AWS C3-4XL instances
○ S3 sink: ~7000 Containers
○ Consumer Kafka sink: ~ 4500 Containers
○ Elasticsearch sink: ~1500 Containers

42. Routing Layer
● Total of ~1400 streams across all regions
○ ~1000 S3 streams
○ ~250 Consumer Kafka streams
○ ~150 Elasticsearch streams

43. Router Job Details
● One Job per sink and Kafka source topic
○ Separate Job each for S3, ElasticSearch & Kafka sink
○ Provides better isolation & better QOS
● Batch processed message requests to sinks
● Offset checkpointed after batch request succeeds

44. Processing Semantics
Data Loss & Duplicates

45. Backpressure
Producer ⇐ Kafka Cluster ⇐ Samza job router ⇐ Sink
● Keystone - at least once

46. Data Loss - Producer
● buffer full
● network error
● partition leader change
● partition migration

47. Data Loss - Kafka
● Lose all Kafka replicas of data
○ Safe guards:
■ AZ isolation / Alerts / Broker replacement automation
■ alerts and monitoring
● Unclean partition leader election
○ ack = 1 could cause loss

48. Data Loss - Router
● Lose checkpointed offset & the router was down for retention period duration
● If messages not processed past retention period (8h / 24h)
● Unclean leader election cause offset to go back
● Safe guard
○ alerts for lag > 0.1% of traffic for 10 minutes
● Concerned only if unable to launch router instances

49. Duplicates Router - Sink
● Duplicates possible
○ messages reprocessed - retry after batch S3 upload failure
○ Loss of checkpointed offset (message processed marker)
○ Event GUID helps dedup

50. Measure Duplicates
● Producer sent count diff with Kafka message received
● Router checkpointed offset monitored over time
Note: GUID can be used to dedup at the sink

52. End to End metrics
● Producer to Router to Sink Average Latencies
○ Batch processing [S3 sink]: ~3 sec
○ Stream processing [Consumer Kafka sink]: ~1 sec
○ Log analysis [Elasticsearch]: ~400 seconds (with back pressure)

53. End to End metrics
● End to End latencies
○ S3:
■ 50 percentile under 1 sec
■ 80 percentile under 8 seconds
○ Consumer Kafka:
■ 50 percentile under 800 ms
■ 80 percentile under 4 seconds
○ Elasticsearch:
■ 50 percentile under 13 seconds
■ 80 percentile under 53 seconds

54. Alerts
● Producer drop rate over 1%
● Consumer lag > 0.1%
● Next offset after Checkpointed offset not found
● Consumer stuck on partition level

55. Keystone Dashboard

56. Keystone Dashboard

57. Keystone Dashboard

58. Keystone Dashboard

59. And Then???

60. There’s more in the pipeline...
● Self service tools
● Better management of scaling Kafka
● More capable control plane
● JSON Support exists, support for Avro on the horizon
● Multi-tenant Messaging as a Service - MaaS
● Multi-tenant Stream Processing as a Service - SPaaS

61. ???s