Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud"
•
4 likes•2,782 views
Over 109 million subscribers are enjoying more than 125 million hours of TV shows and movies per day on Netflix. This leads to massive amount of data flowing through our data ingestion pipeline to improve service and user experience. They are powering various data analytic cases like personalization, operational insight, fraud detection. At the heart of this massive data ingestion pipeline is a self-serve stream processing platform that processes 3 trillion events and 12 PB of data every day. We have recently migrated this stream processing platform from Samza to Flink. In this talk, we will share the challenges and issues that we run into when running Flink at scale in cloud. We will dive deep into the troubleshooting techniques and lessons learned.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud"
Similar to Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud" (20)
More from Flink Forward
More from Flink Forward (12)
Recently uploaded
Recently uploaded (20)
Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud"
- 1. Scaling Flink in Cloud Steven Wu @stevenzwu
- 2. Agenda ● Introduction ● Scaling stateless jobs ● Scaling stateful jobs
- 3. Agenda ● Introduction ● Scaling stateless jobs ● Scaling stateful jobs
- 4. Running Flink on Titus (Netflix’s in-house container runtime)
- 5. Job isolation: single job Job Manager Task Manager Task Manager Task Manager ... Titus Job #1 Titus Job #2 Flink standalone cluster
- 6. State backend and checkpoint store State backend ● Memory ● File system ● RocksDB Source: http://flink.apache.org/ checkpoint store ● HDFS ● S3
- 7. Why S3 as the snapshot store ● Only out-of-the-box support for Amazon cloud ● Cost-effective, scalability, durability
- 8. S3 concepts ● Massive storage system ● Bucket: container for objects ● Object: identified by a key (and a version) ● Filesystem like operations ○ GET, PUT, DELETE, LIST, HEAD
- 9. examplebucket/2018-04-01/00/data1.avro examplebucket/2018-04-01/00/data2.avro examplebucket/2018-04-02/01/data1.avro examplebucket/2018-04-02/01/data2.avro examplebucket/2018-04-02/03/data1.avro examplebucket/2018-04-02/08/data1.avro examplebucket/2018-04-03/23/data1.avro S3 sharding: range partition Partition 1 Partition 2 Partition 3 date / hour / file
- 10. examplebucket/2018-04-01/00/data1.avro examplebucket/2018-04-01/00/data2.avro examplebucket/2018-04-02/01/data1.avro examplebucket/2018-04-02/01/data2.avro examplebucket/2018-04-02/03/data1.avro examplebucket/2018-04-02/08/data1.avro examplebucket/2018-04-03/23/data1.avro LIST (prefix query) Partition 1 Partition 2 Partition 3 date / hour / file
- 11. S3 scaling ● If request rate grows steadily, S3 automatically partitions buckets as needed to support higher request rates
- 12. examplebucket/2018-04-01/00/data1.avro examplebucket/2018-04-01/00/data2.avro examplebucket/2018-04-01/00/data3.avro examplebucket/2018-04-01/00/data4.avro examplebucket/2018-04-01/00/data5.avro examplebucket/2018-04-01/00/data6.avro examplebucket/2018-04-01/00/data7.avro Avoid sequential key names if over 100 reqs/second
- 14. S3 Performance ● Optimized for high I/O throughput ● Not optimized for high request rate without tweaking key names ● Not optimized for small files ● Not optimized for consistent low latency
- 15. Agenda ● Introduction ● Scaling stateless jobs ● Scaling stateful jobs
- 16. Event Producers Sinks highly available ingest pipelines - the backbone of a real-time data infrastructure
- 17. Events are published to fronting Kafka directly or via proxy KSGateway Stream Consumers Event Producer Keystone Management Fronting Kafka Flink Router Consumer Kafka HTTP / gRPC
- 18. Events land up in fronting Kafka cluster KSGateway Stream Consumers Event Producer Keystone Management Fronting Kafka Flink Router Consumer Kafka HTTP / gRPC
- 19. Events are polled by router, filter and projection applied KSGateway Stream Consumers Keystone Management Fronting Kafka Flink Router Consumer Kafka Event Producer HTTP / gRPC
- 20. Router sends events to destination KSGateway Stream Consumers Keystone Management Fronting Kafka Flink Router Consumer Kafka Event Producer HTTP / gRPC
- 22. Keystone routing jobs ● Stateless ● Embarrassingly parallel
- 23. Keystone router scale ● ~3 trillion events/day ● ~2,000 routing jobs ● ~10,000 containers ● ~200,000 parallel operator instances
- 24. Math 101: S3 writes ● ~2,000 routing jobs ● checkpoint interval is 30 seconds ● ~67 (= 2,000 / 30) S3 writes per second?
- 25. Adapted from http://flink.apache.org/ Each operator writes to S3 S3 State Snapshot barriers
- 26. Math 201: S3 writes ● ~200,000 operators. Each operator writes checkpoint to S3 ● checkpoint interval is 30 seconds ● ~6,600 writes (= 200,000 / 30) per second ○ Actual writes 2-3x smaller because only Kafka source operators have state
- 27. S3 throttling!
- 28. S3 not optimized for high request rate without tweaking key names
- 30. Introduce entropy in checkpoint path state.checkpoints.dir:
- 31. S3 not optimized for small files
- 32. Checkpoint ack with metadata Job Manager ACK (metadata) S3 State Snapshot barriers
- 33. Uber checkpoint file after all ACKs Job Manager S3 Uber file with metadata
- 35. Checkpoint ack with state Job Manager S3 barriers ACK (state)
- 36. Uber checkpoint file after all ACKs Job Manager S3 Uber file with metadata + state
- 38. Avoid S3 writes from task managers ● Only job manager writes one uber checkpoint file ● Reduced checkpoint duration by 10x
- 39. S3 HEADs are 100x of PUTs
- 40. Issue #1 Hadoop S3 file system ● Half of the HEADs failed for non-exist objects ● Always two HEADs for the same object (with and without trailing slash) ○ checkpoints/<flink job>/fe68ab5591614163c19b55ff4aa66ac ○ checkpoints/<flink job>/fe68ab5591614163c19b55ff4aa66ac/
- 41. HEAD requests coming from task managers
- 42. BTrace: dynamic tracing tool for Java ● Dynamically trace a running Java process ● Dynamically instruments the classes of the target application to inject tracing code ("bytecode tracing")
- 43. import com.sun.btrace.annotations.*; import static com.sun.btrace.BTraceUtils.*; import static com.sun.btrace.BTraceUtils.Sys.*; import java.util.concurrent.atomic.*; @BTrace public class S3ReqTracing { private static AtomicLong putCounter = newAtomicLong(0); private static AtomicLong headCounter = newAtomicLong(0); @OnMethod( clazz = "com.amazonaws.services.s3.AmazonS3Client", method = "putObject" ) public static void trackPut() { addAndGet(putCounter, 1L); }
- 44. @OnMethod( clazz = "com.amazonaws.services.s3.AmazonS3Client", method = "getObjectMetadata", location = @Location(value=Kind.LINE, line=966) ) public static void trackHead() { addAndGet(headCounter, 1L); jstack(); } @OnTimer(30000) public static void dumpCounters() { printNumber("put", getAndSet(putCounter, 0)); printNumber("head", getAndSet(headCounter, 0)); } }
- 45. Run btrace on task manager ● bin/btrace <PID> S3ReqTracing.java
- 46. Setup ● 1-CPU container ● 1 subtask with two operators
- 47. Findings from task manager ● No S3 writes ● 4 HEAD requests per checkpoint interval ○ 1 (subtask) * 2 (operators) * 2 (with and without trailing slash)
- 49. Math 301: metadata reqs ● ~200,000 operators ● Each operator creates 2 HEAD requests (with and without trailing slash) ● checkpoint interval is 30 seconds ● ~13,000 (200,000 * 2 / 30) HEAD reqs/s from task managers even though they write zero S3 files
- 50. Create CheckpointStreamFactory only once during operator initialization (FLINK-5800) Fixed in 1.2.1 (https://github.com/apache/flink/pull/3312)
- 51. Fine grained recovery (FLIP-1)
- 52. What is fine grained recovery A1 A2 A3 B1 B2 B3 C1 C2 C3
- 53. What is fine grained recovery A1 A2 A3 B1 B2 B3 C1 C2 C3
- 54. What is fine grained recovery A1 A2 A3 B1 B2 B3 C1 C2 C3
- 55. What is fine grained recovery A1 A2 A3 B1 B2 B3 C1 C2 C3
- 56. Life without fine grained recovery Each kill (every 10 minutes) caused ~2x spikes
- 57. Sometimes revert to full restart Full restartFine grained recovery
- 58. Current implementation issue (FLINK-8042) ● Revert to full restart immediately if replacement container didn’t come back in time (FLINK-8042) ● Fix expected in FLIP-6
- 59. Workaround: +1 standby container Job Manager Task Manager Task Manager Task Manager ... Task Manager +1
- 60. Fine grained recovery in action
- 61. Recap of scaling stateless jobs ● Introduce random prefix in checkpoint path to spread S3 writes from many different jobs ● Avoid S3 writes from task managers ● Enable fine grained recovery (+1 standby)
- 62. Agenda ● Introduction ● Scaling stateless jobs ● Scaling stateful jobs
- 63. Often come with data shuffling A1 A2 A3 B1 B2 B3 C1 C2 C3 keyBysource window sink
- 64. Challenges of large-state job ● Introduce random hex chars in checkpoint path to spread S3 writes from different jobs ○ Single job writes large state to S3 ● Avoid S3 writes from task managers ○ Each task manager has large state ● Enable fine grained recovery (+1 standby) ○ Connected job graph
- 65. Challenges of large-state job ● Single job writes large state to S3 ● Each task manager has large state ● Connected job graph
- 66. Inject dynamic entropy in S3 path state.backend.fs.checkpointdir.injectEntropy.enabled: state.backend.fs.checkpointdir.injectEntropy.key: state.checkpoints.dir: 2.5x throughput improvement
- 68. Challenges of large-state job ● Single job writes large state to S3 ● Each task manager has large state ● Connected job graph
- 69. Tuning Flink to stabilize ● Enable incremental checkpoint with RocksDB ● RocksDB tuning: FLASH_SSD_OPTIMIZED ● Network buffer: taskmanager.network.memory.max=4 GB
- 70. Setup for performance test ● Cluster size: 200 nodes ○ 16 CPUs ○ 54 GB memory ○ 108 GB SSD-backed EBS volume ● Parallelism: 3,200
- 71. Numbers for savepoint ● Size: 21 TBs ● Take time: 27 minutes ● Recovery time: 6 minutes
- 72. Numbers for incremental checkpoint ● Checkpoint interval: 15 mins ● Size (avg): ● Duration (avg): 2.5 minutes
- 73. Challenges of large-state job ● Single job writes large state to S3 ● Each task manager has large state ● Connected job graph
- 74. Full restart with connected graph A1 A2 A3 B1 B2 B3 C1 C2 C3
- 75. Full restart with connected graph A1 A2 A3 B1 B2 B3 C1 C2 C3
- 76. Full restart with connected graph A1 A2 A3 B1 B2 B3 C1 C2 C3
- 77. Recover data from S3 A1 A2 A3 B1 B2 B3 C1 C2 C3 TM #1 TM #2 S3 HDD HDD HDDTM #3
- 78. Recover data from S3 A1 A2 A3 B1 B2 B3 C1 C2 C3 TM #1 TM #2 S3 HDD HDD HDDTM #3
- 79. Task local recovery (FLINK-8360) A1 A2 A3 B1 B2 B3 C1 C2 C3 TM #1 TM #2 S3 HDD HDD HDDTM #3
- 80. Task local recovery with EBS A1 A2 A3 B1 B2 B3 C1 C2 C3 TM #1 TM #2 S3 EBS EBS EBSTM #3
- 81. Recap of scaling stateful jobs ● Inject dynamic random prefix in checkpoint path to spread S3 writes from operators in the same job ● Enable incremental checkpoint with RocksDB ● Challenge: connected graph makes recovery more expensive
- 82. Thank you! Steven Wu @stevenzwu