Unified Stream Processing at Scale with Apache Samza - BDS2017

1. 1 Unified Stream Processing at Scale with Apache Samza Jake Maes Staff SW Engineer at LinkedIn Apache Samza PMC
2. 2 Agenda Intro to Stream Processing Stream Processing Ecosystem at LinkedIn Use Case: Pre-Existing Online Service Use Case: Batch  Streaming Future
4. 4 About ● Stream processing framework ● Production at LinkedIn since 2014 ● Apache top level project since 2014 ● 16 Committers ● 74 Contributors ● Known for  Scale  Managed local state  Pluggability  Kafka integration
5. 5 ● Low latency ● One message at a time ● Checkpointing, durable state ● All I/O with high-performance message brokers Traditional Stream Processing
6. 6 Partitioned Processing TaskTask0 State0 Changelog Stream (partition 0) Checkpoint Stream Processor Output StreamsInput Streams (partition 0)
8. 8 ● Anti abuse ● Derived data ● Search Indexing ● Geographic filtering ● A/B testing infrastructure ● Many many more… Stream Processing Use Cases at LinkedIn
9. 9 Stream Processing Ecosystem – The Dream Applications and Services Samza Kafka Storage External Streams Storage & Serving Brooklin
10. 10 Stream Processing Ecosystem - Reality Applications and Services Samza Kafka Storage External Streams Storage & Serving Brooklin
11. 11 Expansion of Stream Processing at LinkedIn ● Influx of applications  10 -> 200+ over 3 years  13K containers processing 260B events/day ● Migrations of existing applications  Online services  Offline jobs ● Incoming applications have different expectations Services
13. 13 Case Study – Notification Scheduler Processor User Chat Event User Action Event Connection Activity Event Restful Services Member profile database Aggregation Engine Channel Selection State store input1 input2 input3 ① Local Data Access ② Remote Database Lookup ③ Remote Service Calloutput
14. 14 Online Service + Stream Processing Why use stream processor? ● Richer framework than Kafka clients Requirements: ● Deployment model  Cluster (YARN) environment not suitable ● Remote I/O  Dependencies on other services  I/O latency stalls single threaded processor  Container parallelism - too much overhead Services
15. 15 App Instance Embedded Samza ● Zookeeper-based JobCoordinator  Uses Zookeeper for leader election  Leader assigns work to the processors ZooKeeper Stream Processor Samza Container Job Coordinator* App Instance Stream Processor Samza Container Job Coordinator App Instance Stream Processor Samza Container Job Coordinator * Leader
16. 16 Asynchronous Event Loop Stream Processor Event Loop  Single thread  1 : Task  n : Task Restful Services Java NIO, Netty
17. 17 Checkpointing ● Sync – Barrier ● Async - Watermark t1 t2 t3 tc t4 checkpoint callback3 complete time callback1 complete callback2 complete callback4 complete
18. 18 Performance for Remote I/O Baseline Thread pool size = 10 Max concurrency = 1 Thread pool size = 10 Max concurrency = 3 Sync I/O with MultithreadingSingle thread
20. 20 Case Study - Unified Metrics with Samza UMP Analyst Pig Script “Compile”Author Generate Fluent Code + Runtime Config Deploy+ +
21. 21 Offline Jobs Why use stream processor? ● Lower latency Requirements: ● HDFS I/O ● Same app in batch and streaming  Best of both worlds ● Composable API
22. 22 Low Level Logic public class PageViewByMemberIdCounterTask implements InitableTask, StreamTask, WindowableTask { private final SystemStream pageViewCounter = new SystemStream("kafka", "MemberPageViews"); private KeyValueStore<String, PageViewPerMemberIdCounterEvent> windowedCounters; private Long windowSize; @Override public void init(Config config, TaskContext context) throws Exception { this.windowedCounters = (KeyValueStore<String, PageViewPerMemberIdCounterEvent>) context.getStore("windowed-counter-store"); this.windowSize = config.getLong("task.window.ms"); } @Override public void window(MessageCollector collector, TaskCoordinator coordinator) throws Exception { getWindowCounterEvent().forEach(counter -> collector.send(new OutgoingMessageEnvelope(pageViewCounter, counter.memberId, counter))); } @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); countPageViewEvent(pve); } }
23. 23 High Level Logic public class RepartitionAndCounterExample implements StreamApplication { @Override public void init(StreamGraph graph, Config config) { MessageStream<PageViewEvent> pve = graph.getInputStream("pageViewEvent", (k, m) -> (PageViewEvent) m); OutputStream<String, MyOutputType, MyOutputType> mpv = graph .getOutputStream("memberPageViews", m -> m.memberId, m -> m); pve .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow(m -> m.memberId, Duration.ofMinutes(5), () -> 0, (m, c) -> c + 1)) .map(MyOutputType::new) .sendTo(mpv); } } Built-in transform functions
24. 24 Batch <-> Streaming streams.pageViewEvent.system=kafka streams.pageViewEvent.physical.name=PageViewEvent streams.memberPageViews.system= kafka streams.memberPageViews.physical.name=MemberPageViews streams.pageViewEvent.system=hdfs streams.pageViewEvent.physical.name=hdfs://mydbsnapshot/PageViewEvent/ streams.memberPageViews.system=hdfs streams.memberPageViews.physical.name=hdfs://myoutputdb/MemberPageViews Streaming config Batch config
25. 25 Performance - HDFS ● Profile count, group by country ● 500 files ● 250GB
26. 26 Agenda Intro to Stream Processing Stream Processing Ecosystem at LinkedIn Use Case: Pre-Existing Service Use Case: Batch  Streaming Future
27. 27 What’s Next? ● SQL  Prototyped 2015  Now getting full time attention ● High Level API extensions  Better config, I/O, windowing, and more ● Beam Runner  Samza performance with Beam API ● Table support
28. 28 Thank You Contact: ● Email dev@samza.apache.org ● Social http://twitter.com/jakemaes Links: ● http://samza.apache.org ● http://github.com/apache/samza ● https://engineering.linkedin.com/blog
29. 29 Bonus Slides
30. 30 High Level API - Composable Operators filter select a subset of messages from the stream map map one input message to an output message flatMap map one input message to 0 or more output messages merge union all inputs into a single output stream partitionBy re-partition the input messages based on a specific field sendTo send the result to an output stream sink send the result to an external system (e.g. external DB) window window aggregation on the input stream join join messages from two input streams Stateless Functions I/O Functions Stateful Functions
31. 31 Co-Partitioned Streams
32. 32 Typical Flow - Two Stages Minimum Re- partition window map sendTo PageVie w Event PageViewEvent ByMemberId PageViewEventP er MemberStream PageViewRepartitionTask PageViewByMemberIdCounterTask

Editor's Notes

Talk is an evolution story of Stream Processing at Linkedin:
Few years ago, services, batch, and stream processing isolated
Now stream processing used everywhere

Talk focuses on LI, but should apply if your organization is looking to adopt or expand its usage of stream processing.
Title of the talk used the word “Unified” = Stream processing framework that can be used by itself, embedded in an online service, or deployed in both streaming and batch environments seamlessly
Latency Spectrum

Samza-Kafka interaction optimized

Performance:
Most processors can handle 10K msg/s per container. Yes, even with state!
Have seen trivial processors like a repartitioner handle as much as 50K msg/s per container
Have run benchmarks showing 1.2M msg/s on a single host
Under the hood:
Partitions
Data parallelism
Could be files on HDFS, partitions in Kafka, etc.
Tasks
Logical unit of execution
Isolated local state
Processor
Computational parallelism (coarse grained, 1 JVM)
Single threaded event loop
Work assignment
Input partitions are assigned to task instances
A particular task instance usually processes 1 partition from each input (topic)
A task instance will often write to all output partitions
Checkpoints are used to track progress
Changelog for state durability
So, how does this fit into the broader ecosystem?

Stream processing center of the world

Left is storage data at rest
Brooklin is stream ingestion normalization layer
CDC plus ingestion from other streams
Events come into Kafka from brooklin or apps & services
Processed by Samza and back out to Kafka
Ingested by other storage and serving components

Common pattern
Optimized for streams (everything is a stream)
Realistic? no
Stream processor is optimized for interacting with streams, it makes sense to pursue an architecture which provides access to all the necessary data sources and sinks as streams.
In reality, streaming applications often need to interface with a number of other systems. Why?
Too expensive to replicate everything into Kafka [Kafka Connect]
Processor was written offline but for latency purposes needs to also run in streaming mode
Some datasources are shared with other services that need Random Access that is easier to provide from a database or serving layer and we don’t want multiple sources of truth
Because some systems don’t have the ability to ingest from a stream (either because it wasn’t created, or they just wouldn’t be able to do it fast enough)
Because sometimes for security purposes, it’s better for an application to interact directly with another

Where does this come from? Well over the remainder of this talk. I’ll describe how stream processing has changed at LinkedIn and dig in to 2 sources of the evolving requirements and how we adapted to them.
As I mentioned earlier, at LinkedIn we’ve been using Samza in production for over 3 years.

In that time it has grown from 10 to over 200 applications.
We now have over 1300 app instances, with an average of 10 containers each, handling over 260B events per day
(conservative numbers

These applications are not all new stream processors; many of them are migrations of existing applications that can be divided into 2 main categories:
Preexis
Why use stream processor?
Abstractions for input output streams, checkpointing, durable state, etc.

Existing services often don’t fit with the YARN deployment model
May already have dedicated hardware they want to use
May require a more static host assignment. e.g. if they’re exposing a RESTful endpoint

Also tend to depend on other services
Datasources with only RESTful or client APIs (not streaming)

Remote I/O introduces huge latency into single-threaded event loop.
Workaround: users would manage their own thread pool and use manual commit in window()
Workaround2: users would just use a massive number of containers to get more parallelism
Metrics used to be run daily with pig scripts
Now same script can be compiled into a Samza processor that runs both online and offline. No need to rewrite for the new platform.
Real time metrics
From a single definition (e.g. a Pig script ) generate batch and real time flows. Flip a config!
How to associate the keys from multiple streams?
Copartitioning

Each input has:
Same partition key
Same partition count
Same partition algorithm (usually hash + modulo)
The result of the co-partitioning requirement:
Most stateful jobs include a repartition stage, which re-keys or reshuffles the inputs to achieve co-partitioning

Often implemented as separate processors that are deployed at the same time.

Unified Stream Processing at Scale with Apache Samza - BDS2017

You are reading a preview.

Check these out next

Unified Stream Processing at Scale with Apache Samza - BDS2017

More Related Content

Slideshows for you (20)

Similar to Unified Stream Processing at Scale with Apache Samza - BDS2017 (20)

Recently uploaded (20)

Unified Stream Processing at Scale with Apache Samza - BDS2017

Editor's Notes

Unified Stream Processing at Scale with Apache Samza - BDS2017

You are reading a preview.

Unified Stream Processing at Scale with Apache Samza - BDS2017

Recommended

More Related Content

Slideshows for you (20)

Similar to Unified Stream Processing at Scale with Apache Samza - BDS2017 (20)

Recently uploaded (20)

Unified Stream Processing at Scale with Apache Samza - BDS2017

Editor's Notes