Downloaded 10 times
Uploaded byBig Data Spain
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data Spain 2017
The document presents an overview of Apache Samza, a stream processing framework utilized at LinkedIn, highlighting its features such as scalability, low latency, and Kafka integration. It covers the stream processing ecosystem at LinkedIn, detailing use cases that integrate pre-existing services, batch and streaming processes, and future enhancements like SQL support and high-level API improvements. Additionally, it discusses performance considerations and architectural components within the context of stream processing applications.
More Related Content
Similar to Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data Spain 2017
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data Spain 2017
- 2. 1 Unified Processing atScale with Apache Samza Jake Maes Staff SW Engineer at LinkedIn Apache Samza PMC
- 3. 2 About Me ● ApacheSamza PMC member ● LinkedIn 3 years ● 8 years performance & infra development ● Passionate about scale ● Long walks on the peaks
- 4. 3 Agenda Intro to StreamProcessing Stream Processing Ecosystem at LinkedIn Use Case: Pre-Existing Service Use Case: Batch Streaming Future
- 5. 4 Agenda Intro to StreamProcessing Stream Processing Ecosystem at LinkedIn Use Case: Pre-Existing Service Use Case: Batch Streaming Future
- 6. 5 About ● Production atLinkedIn since 2014 ● Apache top level project since 2014 ● 16 Committers ● 74 Contributors ● Known for Scale Pluggability Kafka integration
- 7. 6 ● Low latency ●One message at a time ● Checkpointing, state, durability ● All I/O with high-performance message brokers Traditional Stream Processing
- 8.
- 9.
- 10. 9 Typical Flow -Two Stages Minimum Re- partitio n windo w ma p sendT o PageVie w Event PageViewEven t ByMemberId PageViewEventP er MemberStream PageViewRepartitionTask PageViewByMemberIdCounterTask
- 11. 10 Agenda Intro to StreamProcessing Stream Processing Ecosystem at LinkedIn Use Case: Pre-Existing Service Use Case: Batch Streaming Future
- 12. 11 Stream Processing Ecosystem– The Dream Applications and Services Samz a Kafka Storag e Externa l Stream s Storage & Serving Brooklin
- 13. 12 Stream Processing Ecosystem- Reality Applications and Services Samz a Kafka Storag e Externa l Stream s Storage & Serving Brooklin
- 14. 13 Expansion of StreamProcessing at LinkedIn ● Influx of new applications 10 -> over 200 ● New use cases Batch Streaming Remote I/O Composable API ● Incoming applications have different expectations ● Let’s take a look at two Services
- 15. 14 Agenda Intro to StreamProcessing Stream Processing Ecosystem at LinkedIn Use Case: Pre-Existing Service Use Case: Batch Streaming Future
- 16. 15 Online Service +Stream Processing Requirements: ● Deployment model Cluster environment not suitable ● Remote I/O Dependencies on other services I/O latency stalls single threaded processor Container parallelism - too much overhead Services
- 17. 16 App Instance Embedded Samza ●Zookeeper-based JobCoordinator Uses Zookeeper for leader election Leader assigns work to the processors ZooKeeperZooKeeper Stream Processor Samza Container Job Coordinato r* App Instance Stream Processor Samza Container Job Coordinato r App Instance Stream Processor Samza Container Job Coordinato r * Leader
- 18. 17 Asynchronous Event Loop Stream Processor EventLoop Single thread 1 : Task n : Task Restful Services Java NIO, Netty
- 19. 18 Checkpointing ● Sync –Barrier ● Async - Watermark t1 t2 t3 tc t4 checkpoint callback 3 complet e time callback 1 complet e callback 2compl ete callback 4 complet e
- 20. 19 Performance for RemoteI/O Baseline Thread pool size = 10 Max concurrency = 1 Thread pool size = 10 Max concurrency = 3 Sync I/O with MultithreadingSingle thread
- 21. 20 Case Study –Notification Scheduler Processor User Chat Event User Action Event Connectio n Activity Event Restful Service s Member profile database Aggregatio n Engine Channel Selection State store input1 input2 input3 ① Local Data Access ② Remote Database Lookup ③ Remote Service Call outp ut
- 22. 21 Agenda Intro to StreamProcessing Stream Processing Ecosystem at LinkedIn Use Case: Pre-Existing Service Use Case: Batch Streaming Future
- 23. 22 Offline Jobs Requirements: ● Performanceand low latency ● Resource hungry Finite jobs can hog resources Infinite jobs need to be better citizens ● Composable API ● Same app in batch and streaming Best of both worlds ● HDFS I/O
- 24. 23 Low Level Logic publicclass PageViewByMemberIdCounterTask implements InitableTask, StreamTask, WindowableTask { private final SystemStream pageViewCounter = new SystemStream("kafka", "MemberPageViews"); private KeyValueStore<String, PageViewPerMemberIdCounterEvent> windowedCounters; private Long windowSize; @Override public void init(Config config, TaskContext context) throws Exception { this.windowedCounters = (KeyValueStore<String, PageViewPerMemberIdCounterEvent>) context.getStore("windowed-counter-store"); this.windowSize = config.getLong("task.window.ms"); } @Override public void window(MessageCollector collector, TaskCoordinator coordinator) throws Exception { getWindowCounterEvent().forEach(counter -> collector.send(new OutgoingMessageEnvelope(pageViewCounter, counter.memberId, counter))); } @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); countPageViewEvent(pve); } }
- 25. 24 High Level Logic publicclass RepartitionAndCounterExample implements StreamApplication { @Override public void init(StreamGraph graph, Config config) { MessageStream<PageViewEvent> pve = graph.getInputStream("pageViewEvent", (k, m) -> (PageViewEvent) m); OutputStream<String, MyOutputType, MyOutputType> mpv = graph .getOutputStream("memberPageViews", m -> m.memberId, m -> m); pve .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow(m -> m.memberId, Duration.ofMinutes(5), () -> 0, (m, c) -> c + 1)) .map(MyOutputType::new) .sendTo(mpv); } } Built-in transform functions
- 26. 25 High Level API- Composable Operators filter select a subset of messages from the stream map map one input message to an output message flatMap map one input message to 0 or more output messages merge union all inputs into a single output stream partitionBy re-partition the input messages based on a specific field sendTo send the result to an output stream sink send the result to an external system (e.g. external DB) window window aggregation on the input stream join join messages from two input streams Stateless Functions I/O Function s Stateful Functions
- 27. 26 Batch AND Streaming streams.pageViewEvent.system=kafka streams.pageViewEvent.physical.name=PageViewEvent streams.memberPageViews.system=kafka streams.memberPageViews.physical.name=MemberPageViews streams.pageViewEvent.system=hdfs streams.pageViewEvent.physical.name=hdfs://mydbsnapshot/PageViewEven t/ streams.memberPageViews.system=hdfs streams.memberPageViews.physical.name=hdfs://myoutputdb/MemberPage Views Streaming config Batch config
- 28. 27 Case Study -Unified Metrics with Samza UMP Analyst Pig Script “Compile”Author Generate Fluent Code + Runtime Config Deploy+ +
- 29. 28 Performance - HDFS ●Profile count, group by country ● 500 files ● 250GB
- 30. 29 Agenda Intro to StreamProcessing Stream Processing Ecosystem at LinkedIn Use Case: Pre-Existing Service Use Case: Batch Streaming Future
- 31. 30 What’s Next? ● SQL Prototyped 2015 Now getting full time attention ● High Level API extensions Better config, I/O, windowing, and more ● Beam Runner Samza performance with Beam API ● Table support
- 32. 31 Questions Contact: ● Email: dev@samza.apache.org ●Social: http://twitter.com/jakemaes Links: ● http://samza.apache.org ● http://github.com/apache/samza ● https://engineering.linkedin.com/blog