DataWorks Summit, profile picture
Uploaded byDataWorks Summit
PPTX, PDF3,044 views

Unified Batch & Stream Processing with Apache Samza

The document discusses unified batch and stream processing at LinkedIn using Apache Samza, highlighting its role in various data processing scenarios, such as real-time analytics and security measures. Apache Samza supports both batch and stream processing through a unified API, allowing for flexible deployment models and efficient state management. It emphasizes the challenges of maintaining separate systems for batch and stream processing and proposes a solution by treating batch processing as a special case of streaming.

Embed presentation

Downloaded 63 times
Unified Batch & Stream Processing with Apache Samza Navina Ramesh Sr. Software Engineer, LinkedIn Committer and PMC member, Apache Samza @navina_r navina@apache.org
Agenda ● Data Processing at LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
Data Processing at LinkedIn Azure EventHub Oracle DB Espresso DB (NoSQL Store for all user data) Brooklin (DB Change Capture) HDFS Hadoop (Batch Processing) Import / Export Services Tier Ingestion Processing Voldemort / Venice (K-V Store for Derived Data) Samza (Stream Processing) Amazon Kinesis
Scale of Processing at LinkedIn KAFKA 2.3 Trillion Msgs per Day 0.6 PB in, 2.3 PB out per Day (compressed) 16 million Msgs per Second at peaks! HADOOP 125 TB Ingested per Day 120 PB Hdfs Size 200K Jobs per Day SAMZA 200+ Applications Most Applications require Stateful Processing ~ several TBs (overall)
Data Processing Scenarios at LinkedIn Site Speed Real-time site- speed profiling by facets Call-graph Computation Analysis of Service calls Dashboards Real-time Analytics Ad CTR Computation Tracking Ads Views and Ads Clicks Operate primarily using real-time input data
Data Processing Scenarios at LinkedIn News Classification Real-time topic tagging of articles Profile Standardization Standardizing titles, gender, education Security Real-time DDoS protection for members ● Operate on real-time data & rely on models computed offline ● Offline computed model must be accessible during real-time processing
Agenda ● Data Processing at LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
Ingestion Service HDFS Mappers Reducers HDFS/ HBase Processors Processors KV Store Partition 0 Partition 1 Partition N ... Query Data Pipelines in Batch & Stream Azure EventHub Batch / Offline Stream / Realtime Streams to Batch & Batch to Stream
Batch ● Processing on bounded data ● Processing at regular intervals ● Latency ~ order of hours ● Processing on unbounded data ● Processing is continuous ● Latency ~ order of sub-seconds ● Time matters! Stream
● Overhead of developing and managing multiple source codes ○ Same application logic written using 2 different APIs - one using offline processing APIs and another using near-realtime processing API ● Same application deployed in potentially 2 different managed platforms ○ Restrictions due to firewalls, acl to environments etc. ● Expensive $$ ○ When near-realtime application needs processed data from offline, the data snapshot has to be made available as a stream. This is expensive! Data Pipelines in Batch & Stream - Drawbacks
Ingestion Service HDFS Mappers Reducers HDFS/ HBase Processors Processors HDFS KV Store Partition 0 Partition 1 Partition N ... Query Query Data Pipelines in Batch & Stream Azure EventHub Data Sources Data Processing Sink / Serving Batch / Offline Stream / Realtime
Ingestion Service HDFS Mappers Reducers HDFS/ HBase Processors Processors HDFS KV Store Partition 0 Partition 1 Partition N ... Query Query Azure EventHub Data Sources Data Processing Sink / Serving Batch / Offline Stream / Realtime Converge Pipelines with Apache Samza
Agenda ● Data Processing at LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
Apache Samza • Production at LinkedIn since 2013 • Apache TLP since 2014 • Streams as first-class citizen – Batch as a special case of streaming
Apache Samza ● Provides distributed and scalable data processing platform with ○ Configurable and heterogeneous data sources and sinks (Eg. Kafka, HDFS, Kinesis, EventHub etc) ○ Efficient state management - local state and incremental checkpoints ○ Unified Processing API for Batch & Streaming ○ Flexible deployment models
Apache Samza Azure EventHub Amazon Kinesis HDFS Remote Runner Standalone Local Runner AzureYARN Mesos System (Producer& Consumer) Local State (Rocks DB, In-Memory) Checkpoint Manager Remote Data (Multithreading) High-level API Low-level API PROCESSOR DEPLOYM ENT API SQL DB Streams Batch Change Data Capture
Data Processing Model • Natively supports partitioned data • Re-partitioning may be required for an un-partitioned source • Pluggable System and CheckpointManager implementations
Partitions Partitioned Input Tasks 1 2 3 Processing Kafka/Eventhub Client Send with PartitionKey Samza Application - is a made up of Tasks - every Task processes a unique collection of input partitions 1 2 3 4 5 Processing Partitioned Data Single JVM (container)
Partitions Partitioned Input Tasks 1 2 3 Processing Kafka/Eventhub Client Send with PartitionKey - Samza master distributes tasks across JVMs - Scale up & Distribute – increasing container count 1 2 3 4 5 Processing Partitioned Data Distributed across 3 JVMs
Ad View Stream Samza Application 1 2 3 Ad Click Stream Ad Click Through Rate Stream Tasks Processing Joining Co-partitioned Data 1 2 3 1 2 3 Co-partitioned by Ad-ID
Ad View Stream Samza Application 1 2 3 Ad Click Stream Ad Click Through Rate Stream Tasks Processing Joining Co-partitioned Data Local State Store (RocksDB) 1 2 3 1 2 3 Co-partitioned by Ad-ID
Ad View Stream Samza Application 1 2 3 Ad Click Stream Ad Click Through Rate Stream Tasks Processing Joining Co-partitioned Data 1 2 3 1 2 3 Co-partitioned by Ad-ID Changelog Stream for Replication (partitioned) Used for Recovery upon Task Failure
Agenda ● Data Processing at LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
❏Support for Bounded Data ❏ Define a boundary over the stream ❏ Batched Processing ❏Unified Data Processing API ❏Flexible Deployment Models – Write once, Run anywhere! How to converge?
Agenda ● Data Processing at LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
Support for Batch Data • Batch as a special Case of Stream:  Define boundary on stream  Batched processing – end of batch basically ends the job
Defining a Boundary on the Stream • Introduced a notion of End-of-Stream (EoS) in the input • Consumer in the System detects the EoS for a source – Upon EoS, Samza may invoke EndOfStreamListenerTask handler implemented by the application (optional)
File Partitions Partitioned Input Tasks 1 2 3 Processing HDFS Client Store Partitioned Data 1 2 3 4 5 Processing Bounded Data - Single File as a Partition - Directory of Files as a Stream
File Partitions Partitioned Input Tasks 1 2 3 Processing HDFS Client Store Partitioned Data 1 2 3 4 5 Processing Bounded Data - Group of Files as a Partition - Groups defined using GroupingPattern Regex
Support for Batch Data • Batch as a special Case of Stream:  Define boundary on stream  Batched processing – end of batch basically ends the job
Processing Bounded Data 1 2 3 4 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1
1 2 3 4 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1 Processing Bounded Data Samza’s SystemConsumer detects EoS for Partition 1 - doesn’t shut-down the task yet.
1 2 3 4 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1 Processing Bounded Data Continues Processing Partition-0
1 2 3 4 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1 Samza detects EoS for the Partition 1 - shuts-down the task. Processing Bounded Data
1 2 3 4 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1 Task has stopped processing Processing Bounded Data
1 2 3 4 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1 When all Tasks in JVM finish processing, Samza job itself shuts-down. Processing Bounded Data
Batch as a Special Case of Stream  Support Bounded nature of data  Define a boundary over the stream  Processing at regular intervals  Tasks exit upon complete consumption of the batch
Profile count, group-by country 500 files 250GB input Samza HDFS Benchmark * Multiple threads per container *
Agenda ● Data Processing at LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
Example Application Count PageViewEvent for each mobile Device OS in a 5 minute window and send the counts to PageViewEventPerDeviceOS PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo
Samza Low-level API public interface StreamTask { void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) { // process message } } PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo Job 1: PageViewRepartitionTask Job 2: PageViewByDeviceOSCounterTask PageViewEventByDeviceOS
Application using Low-level API public class PageViewRepartitionTask implements StreamTask { private final SystemStream pageViewEventByDeviceOSStream = new SystemStream("kafka", "PaveViewEventByDeviceOS"); @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); collector.send(new OutgoingMessageEnvelope(pageViewEventByDeviceOSStream, pve.memberId, pve)); } } Job-1: Filter & Repartition Job PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo PageViewEventByDeviceOS
Application using Low-level API public class PageViewByDeviceOSCounterTask implements InitableTask, StreamTask, WindowableTask { private final SystemStream pageViewCounterStream = new SystemStream("kafka", "PageViewCountPerDeviceOS"); private KeyValueStore<String, PageViewPerMemberIdCounterEvent> windowedCounters; private Long windowSize; @Override public void init(Config config, TaskContext context) throws Exception { this.windowedCounters = (KeyValueStore<String, PageViewPerMemberIdCounterEvent>) context.getStore("windowed-counter-store"); this.windowSize = config.getLong("task.window.ms"); } @Override public void window(MessageCollector collector, TaskCoordinator coordinator) throws Exception { getWindowCounterEvent().forEach(counter -> collector.send(new OutgoingMessageEnvelope(pageViewCounterStream, counter.memberId, counter))); } @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); countPageViewEvent(pve); } } Job-2: Window-based Counter PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo PageViewEventByDeviceOS
Application using Low-level API public class PageViewByDeviceOSCounterTask implements InitableTask, StreamTask, WindowableTask { private final SystemStream pageViewCounterStream = new SystemStream("kafka", "PageViewCountPerDeviceOS"); private KeyValueStore<String, PageViewPerMemberIdCounterEvent> windowedCounters; private Long windowSize; @Override public void init(Config config, TaskContext context) throws Exception { this.windowedCounters = (KeyValueStore<String, PageViewPerMemberIdCounterEvent>) context.getStore("windowed-counter-store"); this.windowSize = config.getLong("task.window.ms"); } @Override public void window(MessageCollector collector, TaskCoordinator coordinator) throws Exception { getWindowCounterEvent().forEach(counter -> collector.send(new OutgoingMessageEnvelope(pageViewCounterStream, counter.memberId, counter))); } @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); countPageViewEvent(pve); } } ... void countPageViewEvent(PageViewEvent pve) { String key = String.format("%08d-%s", (pve.timestamp - pve.timestamp % this.windowSize), pve.memberId); PageViewPerMemberIdCounterEvent counter = this.windowedCounters.get(key); if (counter == null) { counter = new PageViewPerMemberIdCounterEvent(pve.memberId, (pve.timestamp - pve.timestamp % this.windowSize), 0); } counter.count ++; this.windowedCounters.put(key, counter); } ... Job-2: Window-based Counter PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo PageViewEventByDeviceOS
Application using Low-level API public class PageViewByDeviceOSCounterTask implements InitableTask, StreamTask, WindowableTask { private final SystemStream pageViewCounterStream = new SystemStream("kafka", "PageViewCountPerDeviceOS"); private KeyValueStore<String, PageViewPerMemberIdCounterEvent> windowedCounters; private Long windowSize; @Override public void init(Config config, TaskContext context) throws Exception { this.windowedCounters = (KeyValueStore<String, PageViewPerMemberIdCounterEvent>) context.getStore("windowed-counter-store"); this.windowSize = config.getLong("task.window.ms"); } @Override public void window(MessageCollector collector, TaskCoordinator coordinator) throws Exception { getWindowCounterEvent().forEach(counter -> collector.send(new OutgoingMessageEnvelope(pageViewCounterStream, counter.memberId, counter))); } @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); countPageViewEvent(pve); } } ... List<PageViewPerMemberIdCounterEvent> getWindowCounterEvent() { List<PageViewPerMemberIdCounterEvent> retList = new ArrayList<>(); Long currentTimestamp = System.currentTimeMillis(); Long cutoffTimestamp = currentTimestamp - this.windowSize; String lowerBound = String.format("%08d-", cutoffTimestamp); String upperBound = String.format("%08d-", currentTimestamp + 1); this.windowedCounters.range(lowerBound, upperBound).forEachRemaining(entry -> retList.add(entry.getValue())); return retList; } ... Job-2: Window-based Counter PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo PageViewEventByDeviceOS
Samza High-level API public interface StreamApplication { void init(StreamGraph streamGraph, Config config) { // Process message using DSL- // like declarations } } - Ability to express a multi-stage processing pipeline in a single user program - Built-in library to provide high-level stream transformation functions -> Map, Filter, Window, Partition, Join etc. - Automatically generates the DAG for the application
public class CountByDeviceOSApplication implements StreamApplication { @Override public void init(StreamGraph graph, Config config) { Supplier<Integer> initialValue = () -> 0; MessageStream<PageViewEvent> pageViewEvents = graph.getInputStream("pageViewEvent", (k, m) -> (PageViewEvent) m); OutputStream<String, MyStreamOutput, MyStreamOutput> pageViewEventPerMemberStream = graph .getOutputStream("pageViewCountPerDevice", m -> m.memberId, m -> m); pageViewEvents .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow( m -> m.memberId, Duration.ofMinutes(5),initialValue,(m, c) -> c + 1)) .map(MyStreamOutput::new) .sendTo(pageViewEventPerMemberStream); } } Built-in Transforms Application using High-level API PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo PageViewEventByDeviceOS
public class CountByDeviceOSApplication implements StreamApplication { @Override public void init(StreamGraph graph, Config config) { Supplier<Integer> initialValue = () -> 0; MessageStream<PageViewEvent> pageViewEvents = graph.getInputStream("pageViewEvent", (k, m) -> (PageViewEvent) m); OutputStream<String, MyStreamOutput, MyStreamOutput> pageViewEventPerMemberStream = graph .getOutputStream("pageViewCountPerDevice", m -> m.memberId, m -> m); pageViewEvents .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow( m -> m.memberId, Duration.ofMinutes(5), initialValue, (m, c) -> c + 1)) .map(MyStreamOutput::new) .sendTo(pageViewEventPerMemberStream); } } Unified for Batch & Stream Configuration for Stream Input (Kafka): systems.kafka.samza.factory = org.apache.samza.system.KafkaSystemFactory streams.PageViewEvent.samza.system = kafka streams.PageViewEvent.samza.physical.name = PageViewEvent
public class CountByDeviceOSApplication implements StreamApplication { @Override public void init(StreamGraph graph, Config config) { Supplier<Integer> initialValue = () -> 0; MessageStream<PageViewEvent> pageViewEvents = graph.getInputStream("pageViewEvent", (k, m) -> (PageViewEvent) m); OutputStream<String, MyStreamOutput, MyStreamOutput> pageViewEventPerMemberStream = graph .getOutputStream("pageViewCountPerDevice", m -> m.memberId, m -> m); pageViewEvents .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow( m -> m.memberId, Duration.ofMinutes(5), initialValue, (m, c) -> c + 1)) .map(MyStreamOutput::new) .sendTo(pageViewEventPerMemberStream); } } Unified for Batch & Stream Configuration for Stream Input (Kafka): systems.kafka.samza.factory = org.apache.samza.system.KafkaSystemFactory streams.PageViewEvent.samza.system = kafka streams.PageViewEvent.samza.physical.name = PageViewEvent Configuration for Batch Input (HDFS): systems.hdfs.samza.factory = org.apache.samza.system.HdfsSystemFactory streams.PageViewEvent.samza.system = hdfs streams.PageViewEvent.samza.physical.name = hdfs:/user/nramesh/PageViewEvent
public class CountByDeviceOSApplication implements StreamApplication { @Override public void init(StreamGraph graph, Config config) { Supplier<Integer> initialValue = () -> 0; MessageStream<PageViewEvent> pageViewEvents = graph.getInputStream("pageViewEvent", (k, m) -> (PageViewEvent) m); OutputStream<String, MyStreamOutput, MyStreamOutput> pageViewEventPerMemberStream = graph .getOutputStream("pageViewCountPerDevice", m -> m.memberId, m -> m); pageViewEvents .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow( m -> m.memberId, Duration.ofMinutes(5), initialValue, (m, c) -> c + 1)) .map(MyStreamOutput::new) .sendTo(pageViewEventPerMemberStream); } } Unified for Batch & Stream Configuration for Stream Input (Kafka): systems.kafka.samza.factory = org.apache.samza.system.KafkaSystemFactory streams.PageViewEvent.samza.system = kafka streams.PageViewEvent.samza.physical.name = PageViewEvent Configuration for Batch Input (HDFS): systems.hdfs.samza.factory = org.apache.samza.system.HdfsSystemFactory streams.PageViewEvent.samza.system = hdfs streams.PageViewEvent.samza.physical.name = hdfs:/user/nramesh/PageViewEvent Only Config Change!
High-level API - Visualization for DAG SAMZA Visualizer A visualization of application samza-count-by-device-i001, which consists of 1 job(s), 1 input stream(s), and 1 output stream(s).
High-level API Transforms
Agenda ● Data Processing at LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
Coordination Model • Coordination layer is pluggable in Samza • Samza master / leader – Distributes tasks to processor JVMs – On processor failure, it re-distributes • Available Coordination Mechanisms – Apache Yarn • ApplicationMaster is the leader – Apache Zookeeper • One of the processors is the leader and co-ordinates via Zookeeper – Microsoft Azure • One of the processors is the leader and co-ordinates via Azure’s Blob/Tables Storage
Embedding Processor within Application - An instance of the processor is embedded within user’s application - LocalApplicationRunner helps launch the processor within the application public static void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); CountByDeviceOSApplication app = new CountByDeviceOSApplication(); runner.run(app); runner.waitForFinish(); }
Pluggable Coordination Config public static void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); CountByDeviceOSApplication app = new CountByDeviceOSApplication(); runner.run(app); runner.waitForFinish(); } Configs with Zk-based coordination job.coordinator.factory = org.apache.samza.zk.ZkJobCoordinatorFactory job.coordinator.zk.connect = foobar:2181/samza
Pluggable Coordination Config public static void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); CountByDeviceOSApplication app = new CountByDeviceOSApplication(); runner.run(app); runner.waitForFinish(); } Configs with Azure-based coordination: job.coordinator.factory = org.apache.samza.azure.AzureJobCoordinatorFactory job.coordinator.azure.connect = http://foobar:29892/storage/ Configs with Zk-based coordination job.coordinator.factory = org.apache.samza.zk.ZkJobCoordinatorFactory job.coordinator.zk.connect = foobar:2181/samza
Pluggable Coordination Config public static void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); CountByDeviceOSApplication app = new CountByDeviceOSApplication(); runner.run(app); runner.waitForFinish(); } Only Config Change! Configs with Azure-based coordination: job.coordinator.factory = org.apache.samza.azure.AzureJobCoordinatorFactory job.coordinator.azure.connect = http://foobar:29892/storage/ Configs with Zk-based coordination job.coordinator.factory = org.apache.samza.zk.ZkJobCoordinatorFactory job.coordinator.zk.connect = foobar:2181/samza
Deploying Samza in a Managed Cluster (Yarn) app.class = MyStreamApplication RemoteAppplicationRunner: main() RM NM LocalApplicationRunner StreamProcessor JobCoordinator NM NM LocalApplicationRunner StreamProcessor Client Submits JAR run-jc.sh run-app.sh run-local-app.sh run-local-app.sh
Flexible Deployment Models Samza as a Library - Run embedded stream processing in user program - Use Zookeeper for partition distribution among tasks and liveness of processors - Seamlessly scale by spinning a new processor instance Samza as a Service - Run stream processing as a managed program in a cluster (eg. Yarn) - Works with the cluster manager (Eg. AM/RM) for partition distribution among tasks and liveness of processors - Better for resource sharing in a multi- tenant environment
Conclusion ● Easily Composable Architecture allows varied data source consumption ● Write Once, Run Anywhere paradigm ○ Unified API - application logic to be written only once ○ Pluggable Coordination Model - allows application deployment across different execution environment
Future Work ● Support SQL on Streams with Samza ● Table Abstraction in Samza ● Event-time processing ● Samza runner for Apache Beam Contributions are welcome! ● Contributor’s Corner - http://samza.apache.org/contribute/contributors-corner.html ● Ask any question - dev@samza.apache.org ● Follow or tweet us @apachesamza
Questions?
Extra Slides
Lambda-less Architecture with Samza Profile Updates Kafka stream Standardization Normalized Profile Updates Kafka stream Member Profiles
Lambda-less Architecture with Samza Profile Updates Kafka stream Standardization Normalized Profile Updates Kafka stream Member Profiles update the standardization job
Lambda-less Architecture with Samza Profile Updates Kafka stream Standardization Normalized Profile Updates Kafka stream Member Profiles update the standardization job DB Snapshot Standardization Merge & Store Results
Lambda-less Architecture with Samza Profile Updates Kafka stream Standardization Normalized Profile Updates Kafka stream Member Profiles DB Snapshot Standardization Merge & Store Results Stream Processing Batch Processing
Lambda Architecture with Samza

More Related Content

PDF
Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...
byYael Garten
 
PDF
More Data, More Problems: Scaling Kafka-Mirroring Pipelines at LinkedIn
byconfluent
 
PPTX
Kubernetes 101
byStanislav Pogrebnyak
 
PDF
Conquering the Lambda architecture in LinkedIn metrics platform with Apache C...
byKhai Tran
 
PPTX
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
byFlink Forward
 
PDF
Metal³ – Metal Kubed, Bare Metal Provisioning for Kubernetes | Kim Bảo Long
byVietnam Open Infrastructure User Group
 
PDF
VMware Virtual SAN Presentation
byvirtualsouthwest
 
PDF
What Is Kubernetes | Kubernetes Introduction | Kubernetes Tutorial For Beginn...
byEdureka!
 
Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...
byYael Garten
 
More Data, More Problems: Scaling Kafka-Mirroring Pipelines at LinkedIn
byconfluent
 
Kubernetes 101
byStanislav Pogrebnyak
 
Conquering the Lambda architecture in LinkedIn metrics platform with Apache C...
byKhai Tran
 
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
byFlink Forward
 
Metal³ – Metal Kubed, Bare Metal Provisioning for Kubernetes | Kim Bảo Long
byVietnam Open Infrastructure User Group
 
VMware Virtual SAN Presentation
byvirtualsouthwest
 
What Is Kubernetes | Kubernetes Introduction | Kubernetes Tutorial For Beginn...
byEdureka!
 

What's hot

PDF
Kubernetes Architecture and Introduction
byStefan Schimanski
 
PPTX
Five common customer use cases for Virtual SAN - VMworld US / 2015
byDuncan Epping
 
PDF
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
byKai Wähner
 
PPTX
Introduction to Apache Kafka
byAIMDek Technologies
 
ODP
Stream processing using Kafka
byKnoldus Inc.
 
PDF
Apache Kafka Architecture & Fundamentals Explained
byconfluent
 
PPTX
Observability
byMaganathin Veeraragaloo
 
PPSX
Big Data Redis Mongodb Dynamodb Sharding
byAraf Karsh Hamid
 
PPTX
Microservices Architecture Part 2 Event Sourcing and Saga
byAraf Karsh Hamid
 
PDF
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
byKai Wähner
 
PDF
Kafka Streams: What it is, and how to use it?
byconfluent
 
PDF
Open shift 4 infra deep dive
byWinton Winton
 
PPTX
Ceph Intro and Architectural Overview by Ross Turk
bybuildacloud
 
PPTX
Azure Security and Management
byAllen Brokken
 
PPTX
Microservices Part 3 Service Mesh and Kafka
byAraf Karsh Hamid
 
PPTX
Microsoft Azure Cost Optimization and improve efficiency
byKushan Lahiru Perera
 
PPTX
Azure Container Apps
byKen Sykora
 
PPTX
Introduction to Redis
byArnab Mitra
 
PPTX
Kafka 101
byClement Demonchy
 
PPTX
An Introduction to Confluent Cloud: Apache Kafka as a Service
byconfluent
 
Kubernetes Architecture and Introduction
byStefan Schimanski
 
Five common customer use cases for Virtual SAN - VMworld US / 2015
byDuncan Epping
 
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
byKai Wähner
 
Introduction to Apache Kafka
byAIMDek Technologies
 
Stream processing using Kafka
byKnoldus Inc.
 
Apache Kafka Architecture & Fundamentals Explained
byconfluent
 
Observability
byMaganathin Veeraragaloo
 
Big Data Redis Mongodb Dynamodb Sharding
byAraf Karsh Hamid
 
Microservices Architecture Part 2 Event Sourcing and Saga
byAraf Karsh Hamid
 
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
byKai Wähner
 
Kafka Streams: What it is, and how to use it?
byconfluent
 
Open shift 4 infra deep dive
byWinton Winton
 
Ceph Intro and Architectural Overview by Ross Turk
bybuildacloud
 
Azure Security and Management
byAllen Brokken
 
Microservices Part 3 Service Mesh and Kafka
byAraf Karsh Hamid
 
Microsoft Azure Cost Optimization and improve efficiency
byKushan Lahiru Perera
 
Azure Container Apps
byKen Sykora
 
Introduction to Redis
byArnab Mitra
 
Kafka 101
byClement Demonchy
 
An Introduction to Confluent Cloud: Apache Kafka as a Service
byconfluent
 

Similar to Unified Batch & Stream Processing with Apache Samza

PPTX
Apache samza past, present and future
byEd Yakabosky
 
PDF
Apache Samza Past, Present and Future
byKartik Paramasivam
 
PDF
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...
byBig Data Spain
 
PDF
ApacheCon BigData - What it takes to process a trillion events a day?
byJagadish Venkatraman
 
PPTX
Kafka Summit NYC 2017 - Data Processing at LinkedIn with Apache Kafka
byconfluent
 
PPTX
Samza tech talk_2015 - huawei
byYi Pan
 
PPTX
Samza la hug
bySriram Subramanian
 
PPTX
Samza Demo @scale 2017
byXinyu Liu
 
PPTX
Samza tech talk_2015 - strata
byYi Pan
 
PDF
Unbounded bounded-data-strangeloop-2016-monal-daxini
byMonal Daxini
 
PPTX
Samza 0.13 meetup slide v1.0.pptx
byYi Pan
 
PDF
Scalable Stream Processing with Apache Samza
byPrateek Maheshwari
 
PPTX
Essential Ingredients of Realtime Stream Processing @ Scale
byKartik Paramasivam
 
PDF
Apache samza
byHumberto Streb
 
PPT
Moving Towards a Streaming Architecture
byGabriele Modena
 
PPTX
Samza at LinkedIn: Taking Stream Processing to the Next Level
byMartin Kleppmann
 
PPTX
Will it Scale? The Secrets behind Scaling Stream Processing Applications
byNavina Ramesh
 
PDF
Unified Stream & Batch Processing with Apache Flink (Hadoop Summit Dublin 2016)
byucelebi
 
PDF
Scalable complex event processing on samza @UBER
byShuyi Chen
 
PDF
Essential ingredients for real time stream processing @Scale by Kartik pParam...
byBig Data Spain
 
Apache samza past, present and future
byEd Yakabosky
 
Apache Samza Past, Present and Future
byKartik Paramasivam
 
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...
byBig Data Spain
 
ApacheCon BigData - What it takes to process a trillion events a day?
byJagadish Venkatraman
 
Kafka Summit NYC 2017 - Data Processing at LinkedIn with Apache Kafka
byconfluent
 
Samza tech talk_2015 - huawei
byYi Pan
 
Samza la hug
bySriram Subramanian
 
Samza Demo @scale 2017
byXinyu Liu
 
Samza tech talk_2015 - strata
byYi Pan
 
Unbounded bounded-data-strangeloop-2016-monal-daxini
byMonal Daxini
 
Samza 0.13 meetup slide v1.0.pptx
byYi Pan
 
Scalable Stream Processing with Apache Samza
byPrateek Maheshwari
 
Essential Ingredients of Realtime Stream Processing @ Scale
byKartik Paramasivam
 
Apache samza
byHumberto Streb
 
Moving Towards a Streaming Architecture
byGabriele Modena
 
Samza at LinkedIn: Taking Stream Processing to the Next Level
byMartin Kleppmann
 
Will it Scale? The Secrets behind Scaling Stream Processing Applications
byNavina Ramesh
 
Unified Stream & Batch Processing with Apache Flink (Hadoop Summit Dublin 2016)
byucelebi
 
Scalable complex event processing on samza @UBER
byShuyi Chen
 
Essential ingredients for real time stream processing @Scale by Kartik pParam...
byBig Data Spain
 

More from DataWorks Summit

PPTX
Data Science Crash Course
byDataWorks Summit
 
PPTX
Floating on a RAFT: HBase Durability with Apache Ratis
byDataWorks Summit
 
PPTX
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
byDataWorks Summit
 
PDF
HBase Tales From the Trenches - Short stories about most common HBase operati...
byDataWorks Summit
 
PPTX
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
byDataWorks Summit
 
PPTX
Managing the Dewey Decimal System
byDataWorks Summit
 
PPTX
Practical NoSQL: Accumulo's dirlist Example
byDataWorks Summit
 
PPTX
HBase Global Indexing to support large-scale data ingestion at Uber
byDataWorks Summit
 
PPTX
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
byDataWorks Summit
 
PPTX
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
byDataWorks Summit
 
PPTX
Supporting Apache HBase : Troubleshooting and Supportability Improvements
byDataWorks Summit
 
PPTX
Security Framework for Multitenant Architecture
byDataWorks Summit
 
PDF
Presto: Optimizing Performance of SQL-on-Anything Engine
byDataWorks Summit
 
PPTX
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
byDataWorks Summit
 
PPTX
Extending Twitter's Data Platform to Google Cloud
byDataWorks Summit
 
PPTX
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
byDataWorks Summit
 
PPTX
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
byDataWorks Summit
 
PPTX
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
byDataWorks Summit
 
PDF
Computer Vision: Coming to a Store Near You
byDataWorks Summit
 
PPTX
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
byDataWorks Summit
 
Data Science Crash Course
byDataWorks Summit
 
Floating on a RAFT: HBase Durability with Apache Ratis
byDataWorks Summit
 
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
byDataWorks Summit
 
HBase Tales From the Trenches - Short stories about most common HBase operati...
byDataWorks Summit
 
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
byDataWorks Summit
 
Managing the Dewey Decimal System
byDataWorks Summit
 
Practical NoSQL: Accumulo's dirlist Example
byDataWorks Summit
 
HBase Global Indexing to support large-scale data ingestion at Uber
byDataWorks Summit
 
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
byDataWorks Summit
 
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
byDataWorks Summit
 
Supporting Apache HBase : Troubleshooting and Supportability Improvements
byDataWorks Summit
 
Security Framework for Multitenant Architecture
byDataWorks Summit
 
Presto: Optimizing Performance of SQL-on-Anything Engine
byDataWorks Summit
 
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
byDataWorks Summit
 
Extending Twitter's Data Platform to Google Cloud
byDataWorks Summit
 
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
byDataWorks Summit
 
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
byDataWorks Summit
 
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
byDataWorks Summit
 
Computer Vision: Coming to a Store Near You
byDataWorks Summit
 
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
byDataWorks Summit
 

Recently uploaded

PPTX
2026 SCORM Troubleshooting Rustici + dominKnow.pptx
byRustici Software
 
PPTX
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 2
byDianaGray10
 
PPTX
Retrieval Augmented Generation- The Synergistic Power of Prompt Engineering
bySemantic SEO BD
 
PDF
Analyze and Preserve Logs - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
OpenCharacter AI Reviews: Features, Plans, and Best Alternative
byOpenCharacter AI
 
PPTX
Cyber security chapter 1 for learning and educating to the students who want ...
byGulMohammadi
 
PDF
What Is the Azure AI Foundry and Why Does It Matter for Enterprises?
byjohncarterjn
 
PDF
The Enterprise Web3 Landscape - a view from Kaleido based on the past 10 year...
byPeter Broadhurst
 
PDF
ICT500 - CRITICAL AND CREATIVE THINKING FOR INFORMATION TECHNOLOGY SOLUTIONS:...
by2024432452
 
PDF
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
PPTX
Jisc Equipment Data Service Update Workshop 3 December 2025
byJisc
 
PDF
How to build hackthon projects from ZERO!
byishantyadav1111
 
PDF
Agentic-Document-Extraction-2026-Presentation.pdf
byTamanna
 
PPTX
The Lex Wire Precedent: A Technical Standard for Machine-Mediated Authority ...
byJeff Howell
 
PPTX
Introduction to Artificial Intelligence (AI).pptx
byammar414783
 
PDF
Pneumatic Pressure Pump PPP01 for Calibration & Testing
bySERRAX TECHNOLOGIES LLP
 
PDF
Safer’s Picks: The 5 FME Transformers You Didn’t Know You Needed
bySafe Software
 
PDF
Adapt.com Seed Fundraising Deck | The AI Computer for Business
byashley459565
 
PDF
Post-Hackathon-Learnings-Maximizing-Impact-Beyond-the-Event.pdf
byishantyadav1111
 
2026 SCORM Troubleshooting Rustici + dominKnow.pptx
byRustici Software
 
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
UiPath Automation Developer Associate Training Series 2026 - Session 2
byDianaGray10
 
Retrieval Augmented Generation- The Synergistic Power of Prompt Engineering
bySemantic SEO BD
 
Analyze and Preserve Logs - RHCSA (RH134).pdf
byLinuxCert Guru
 
OpenCharacter AI Reviews: Features, Plans, and Best Alternative
byOpenCharacter AI
 
Cyber security chapter 1 for learning and educating to the students who want ...
byGulMohammadi
 
What Is the Azure AI Foundry and Why Does It Matter for Enterprises?
byjohncarterjn
 
The Enterprise Web3 Landscape - a view from Kaleido based on the past 10 year...
byPeter Broadhurst
 
ICT500 - CRITICAL AND CREATIVE THINKING FOR INFORMATION TECHNOLOGY SOLUTIONS:...
by2024432452
 
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
Jisc Equipment Data Service Update Workshop 3 December 2025
byJisc
 
How to build hackthon projects from ZERO!
byishantyadav1111
 
Agentic-Document-Extraction-2026-Presentation.pdf
byTamanna
 
The Lex Wire Precedent: A Technical Standard for Machine-Mediated Authority ...
byJeff Howell
 
Introduction to Artificial Intelligence (AI).pptx
byammar414783
 
Pneumatic Pressure Pump PPP01 for Calibration & Testing
bySERRAX TECHNOLOGIES LLP
 
Safer’s Picks: The 5 FME Transformers You Didn’t Know You Needed
bySafe Software
 
Adapt.com Seed Fundraising Deck | The AI Computer for Business
byashley459565
 
Post-Hackathon-Learnings-Maximizing-Impact-Beyond-the-Event.pdf
byishantyadav1111
 

Unified Batch & Stream Processing with Apache Samza

  • 1.
    Unified Batch &Stream Processing with Apache Samza Navina Ramesh Sr. Software Engineer, LinkedIn Committer and PMC member, Apache Samza @navina_r navina@apache.org
  • 2.
    Agenda ● Data Processingat LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
  • 3.
    Data Processing atLinkedIn Azure EventHub Oracle DB Espresso DB (NoSQL Store for all user data) Brooklin (DB Change Capture) HDFS Hadoop (Batch Processing) Import / Export Services Tier Ingestion Processing Voldemort / Venice (K-V Store for Derived Data) Samza (Stream Processing) Amazon Kinesis
  • 4.
    Scale of Processingat LinkedIn KAFKA 2.3 Trillion Msgs per Day 0.6 PB in, 2.3 PB out per Day (compressed) 16 million Msgs per Second at peaks! HADOOP 125 TB Ingested per Day 120 PB Hdfs Size 200K Jobs per Day SAMZA 200+ Applications Most Applications require Stateful Processing ~ several TBs (overall)
  • 5.
    Data Processing Scenariosat LinkedIn Site Speed Real-time site- speed profiling by facets Call-graph Computation Analysis of Service calls Dashboards Real-time Analytics Ad CTR Computation Tracking Ads Views and Ads Clicks Operate primarily using real-time input data
  • 6.
    Data Processing Scenariosat LinkedIn News Classification Real-time topic tagging of articles Profile Standardization Standardizing titles, gender, education Security Real-time DDoS protection for members ● Operate on real-time data & rely on models computed offline ● Offline computed model must be accessible during real-time processing
  • 7.
    Agenda ● Data Processingat LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
  • 8.
    Ingestion Service HDFS Mappers Reducers HDFS/ HBase Processors Processors KVStore Partition 0 Partition 1 Partition N ... Query Data Pipelines in Batch & Stream Azure EventHub Batch / Offline Stream / Realtime Streams to Batch & Batch to Stream
  • 9.
    Batch ● Processing onbounded data ● Processing at regular intervals ● Latency ~ order of hours ● Processing on unbounded data ● Processing is continuous ● Latency ~ order of sub-seconds ● Time matters! Stream
  • 10.
    ● Overhead ofdeveloping and managing multiple source codes ○ Same application logic written using 2 different APIs - one using offline processing APIs and another using near-realtime processing API ● Same application deployed in potentially 2 different managed platforms ○ Restrictions due to firewalls, acl to environments etc. ● Expensive $$ ○ When near-realtime application needs processed data from offline, the data snapshot has to be made available as a stream. This is expensive! Data Pipelines in Batch & Stream - Drawbacks
  • 11.
    Ingestion Service HDFS Mappers Reducers HDFS/ HBase Processors Processors HDFS KV Store Partition0 Partition 1 Partition N ... Query Query Data Pipelines in Batch & Stream Azure EventHub Data Sources Data Processing Sink / Serving Batch / Offline Stream / Realtime
  • 12.
    Ingestion Service HDFS Mappers Reducers HDFS/ HBase Processors Processors HDFS KV Store Partition0 Partition 1 Partition N ... Query Query Azure EventHub Data Sources Data Processing Sink / Serving Batch / Offline Stream / Realtime Converge Pipelines with Apache Samza
  • 13.
    Agenda ● Data Processingat LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
  • 14.
    Apache Samza • Productionat LinkedIn since 2013 • Apache TLP since 2014 • Streams as first-class citizen – Batch as a special case of streaming
  • 15.
    Apache Samza ● Providesdistributed and scalable data processing platform with ○ Configurable and heterogeneous data sources and sinks (Eg. Kafka, HDFS, Kinesis, EventHub etc) ○ Efficient state management - local state and incremental checkpoints ○ Unified Processing API for Batch & Streaming ○ Flexible deployment models
  • 16.
    Apache Samza Azure EventHub Amazon Kinesis HDFS Remote Runner Standalone LocalRunner AzureYARN Mesos System (Producer& Consumer) Local State (Rocks DB, In-Memory) Checkpoint Manager Remote Data (Multithreading) High-level API Low-level API PROCESSOR DEPLOYM ENT API SQL DB Streams Batch Change Data Capture
  • 17.
    Data Processing Model •Natively supports partitioned data • Re-partitioning may be required for an un-partitioned source • Pluggable System and CheckpointManager implementations
  • 18.
    Partitions Partitioned Input Tasks 1 2 3 Processing Kafka/Eventhub Client Send withPartitionKey Samza Application - is a made up of Tasks - every Task processes a unique collection of input partitions 1 2 3 4 5 Processing Partitioned Data Single JVM (container)
  • 19.
    Partitions Partitioned Input Tasks 1 2 3 Processing Kafka/Eventhub Client Send withPartitionKey - Samza master distributes tasks across JVMs - Scale up & Distribute – increasing container count 1 2 3 4 5 Processing Partitioned Data Distributed across 3 JVMs
  • 20.
    Ad View Stream SamzaApplication 1 2 3 Ad Click Stream Ad Click Through Rate Stream Tasks Processing Joining Co-partitioned Data 1 2 3 1 2 3 Co-partitioned by Ad-ID
  • 21.
    Ad View Stream SamzaApplication 1 2 3 Ad Click Stream Ad Click Through Rate Stream Tasks Processing Joining Co-partitioned Data Local State Store (RocksDB) 1 2 3 1 2 3 Co-partitioned by Ad-ID
  • 22.
    Ad View Stream SamzaApplication 1 2 3 Ad Click Stream Ad Click Through Rate Stream Tasks Processing Joining Co-partitioned Data 1 2 3 1 2 3 Co-partitioned by Ad-ID Changelog Stream for Replication (partitioned) Used for Recovery upon Task Failure
  • 23.
    Agenda ● Data Processingat LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
  • 24.
    ❏Support for BoundedData ❏ Define a boundary over the stream ❏ Batched Processing ❏Unified Data Processing API ❏Flexible Deployment Models – Write once, Run anywhere! How to converge?
  • 25.
    Agenda ● Data Processingat LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
  • 26.
    Support for BatchData • Batch as a special Case of Stream:  Define boundary on stream  Batched processing – end of batch basically ends the job
  • 27.
    Defining a Boundaryon the Stream • Introduced a notion of End-of-Stream (EoS) in the input • Consumer in the System detects the EoS for a source – Upon EoS, Samza may invoke EndOfStreamListenerTask handler implemented by the application (optional)
  • 28.
    File Partitions Partitioned Input Tasks 1 2 3 Processing HDFS Client StorePartitioned Data 1 2 3 4 5 Processing Bounded Data - Single File as a Partition - Directory of Files as a Stream
  • 29.
    File Partitions Partitioned Input Tasks 1 2 3 Processing HDFS Client StorePartitioned Data 1 2 3 4 5 Processing Bounded Data - Group of Files as a Partition - Groups defined using GroupingPattern Regex
  • 30.
    Support for BatchData • Batch as a special Case of Stream:  Define boundary on stream  Batched processing – end of batch basically ends the job
  • 31.
    Processing Bounded Data 12 3 4 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1
  • 32.
    1 2 34 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1 Processing Bounded Data Samza’s SystemConsumer detects EoS for Partition 1 - doesn’t shut-down the task yet.
  • 33.
    1 2 34 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1 Processing Bounded Data Continues Processing Partition-0
  • 34.
    1 2 34 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1 Samza detects EoS for the Partition 1 - shuts-down the task. Processing Bounded Data
  • 35.
    1 2 34 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1 Task has stopped processing Processing Bounded Data
  • 36.
    1 2 34 5 6 1 2 3 4 5 1 2 3 1 2 3 4 5 Partition 0 Partition 1 Partition 2 Partition 3 Bounded Input Task-0 Task-1 When all Tasks in JVM finish processing, Samza job itself shuts-down. Processing Bounded Data
  • 37.
    Batch as aSpecial Case of Stream  Support Bounded nature of data  Define a boundary over the stream  Processing at regular intervals  Tasks exit upon complete consumption of the batch
  • 38.
    Profile count, group-by country 500files 250GB input Samza HDFS Benchmark * Multiple threads per container *
  • 39.
    Agenda ● Data Processingat LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
  • 40.
    Example Application Count PageViewEventfor each mobile Device OS in a 5 minute window and send the counts to PageViewEventPerDeviceOS PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo
  • 41.
    Samza Low-level API publicinterface StreamTask { void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) { // process message } } PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo Job 1: PageViewRepartitionTask Job 2: PageViewByDeviceOSCounterTask PageViewEventByDeviceOS
  • 42.
    Application using Low-levelAPI public class PageViewRepartitionTask implements StreamTask { private final SystemStream pageViewEventByDeviceOSStream = new SystemStream("kafka", "PaveViewEventByDeviceOS"); @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); collector.send(new OutgoingMessageEnvelope(pageViewEventByDeviceOSStream, pve.memberId, pve)); } } Job-1: Filter & Repartition Job PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo PageViewEventByDeviceOS
  • 43.
    Application using Low-levelAPI public class PageViewByDeviceOSCounterTask implements InitableTask, StreamTask, WindowableTask { private final SystemStream pageViewCounterStream = new SystemStream("kafka", "PageViewCountPerDeviceOS"); private KeyValueStore<String, PageViewPerMemberIdCounterEvent> windowedCounters; private Long windowSize; @Override public void init(Config config, TaskContext context) throws Exception { this.windowedCounters = (KeyValueStore<String, PageViewPerMemberIdCounterEvent>) context.getStore("windowed-counter-store"); this.windowSize = config.getLong("task.window.ms"); } @Override public void window(MessageCollector collector, TaskCoordinator coordinator) throws Exception { getWindowCounterEvent().forEach(counter -> collector.send(new OutgoingMessageEnvelope(pageViewCounterStream, counter.memberId, counter))); } @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); countPageViewEvent(pve); } } Job-2: Window-based Counter PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo PageViewEventByDeviceOS
  • 44.
    Application using Low-levelAPI public class PageViewByDeviceOSCounterTask implements InitableTask, StreamTask, WindowableTask { private final SystemStream pageViewCounterStream = new SystemStream("kafka", "PageViewCountPerDeviceOS"); private KeyValueStore<String, PageViewPerMemberIdCounterEvent> windowedCounters; private Long windowSize; @Override public void init(Config config, TaskContext context) throws Exception { this.windowedCounters = (KeyValueStore<String, PageViewPerMemberIdCounterEvent>) context.getStore("windowed-counter-store"); this.windowSize = config.getLong("task.window.ms"); } @Override public void window(MessageCollector collector, TaskCoordinator coordinator) throws Exception { getWindowCounterEvent().forEach(counter -> collector.send(new OutgoingMessageEnvelope(pageViewCounterStream, counter.memberId, counter))); } @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); countPageViewEvent(pve); } } ... void countPageViewEvent(PageViewEvent pve) { String key = String.format("%08d-%s", (pve.timestamp - pve.timestamp % this.windowSize), pve.memberId); PageViewPerMemberIdCounterEvent counter = this.windowedCounters.get(key); if (counter == null) { counter = new PageViewPerMemberIdCounterEvent(pve.memberId, (pve.timestamp - pve.timestamp % this.windowSize), 0); } counter.count ++; this.windowedCounters.put(key, counter); } ... Job-2: Window-based Counter PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo PageViewEventByDeviceOS
  • 45.
    Application using Low-levelAPI public class PageViewByDeviceOSCounterTask implements InitableTask, StreamTask, WindowableTask { private final SystemStream pageViewCounterStream = new SystemStream("kafka", "PageViewCountPerDeviceOS"); private KeyValueStore<String, PageViewPerMemberIdCounterEvent> windowedCounters; private Long windowSize; @Override public void init(Config config, TaskContext context) throws Exception { this.windowedCounters = (KeyValueStore<String, PageViewPerMemberIdCounterEvent>) context.getStore("windowed-counter-store"); this.windowSize = config.getLong("task.window.ms"); } @Override public void window(MessageCollector collector, TaskCoordinator coordinator) throws Exception { getWindowCounterEvent().forEach(counter -> collector.send(new OutgoingMessageEnvelope(pageViewCounterStream, counter.memberId, counter))); } @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); countPageViewEvent(pve); } } ... List<PageViewPerMemberIdCounterEvent> getWindowCounterEvent() { List<PageViewPerMemberIdCounterEvent> retList = new ArrayList<>(); Long currentTimestamp = System.currentTimeMillis(); Long cutoffTimestamp = currentTimestamp - this.windowSize; String lowerBound = String.format("%08d-", cutoffTimestamp); String upperBound = String.format("%08d-", currentTimestamp + 1); this.windowedCounters.range(lowerBound, upperBound).forEachRemaining(entry -> retList.add(entry.getValue())); return retList; } ... Job-2: Window-based Counter PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo PageViewEventByDeviceOS
  • 46.
    Samza High-level API publicinterface StreamApplication { void init(StreamGraph streamGraph, Config config) { // Process message using DSL- // like declarations } } - Ability to express a multi-stage processing pipeline in a single user program - Built-in library to provide high-level stream transformation functions -> Map, Filter, Window, Partition, Join etc. - Automatically generates the DAG for the application
  • 47.
    public class CountByDeviceOSApplicationimplements StreamApplication { @Override public void init(StreamGraph graph, Config config) { Supplier<Integer> initialValue = () -> 0; MessageStream<PageViewEvent> pageViewEvents = graph.getInputStream("pageViewEvent", (k, m) -> (PageViewEvent) m); OutputStream<String, MyStreamOutput, MyStreamOutput> pageViewEventPerMemberStream = graph .getOutputStream("pageViewCountPerDevice", m -> m.memberId, m -> m); pageViewEvents .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow( m -> m.memberId, Duration.ofMinutes(5),initialValue,(m, c) -> c + 1)) .map(MyStreamOutput::new) .sendTo(pageViewEventPerMemberStream); } } Built-in Transforms Application using High-level API PageViewEvent PageViewCountPerDeviceOS Filter & Re- partition Window Map SendTo PageViewEventByDeviceOS
  • 48.
    public class CountByDeviceOSApplicationimplements StreamApplication { @Override public void init(StreamGraph graph, Config config) { Supplier<Integer> initialValue = () -> 0; MessageStream<PageViewEvent> pageViewEvents = graph.getInputStream("pageViewEvent", (k, m) -> (PageViewEvent) m); OutputStream<String, MyStreamOutput, MyStreamOutput> pageViewEventPerMemberStream = graph .getOutputStream("pageViewCountPerDevice", m -> m.memberId, m -> m); pageViewEvents .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow( m -> m.memberId, Duration.ofMinutes(5), initialValue, (m, c) -> c + 1)) .map(MyStreamOutput::new) .sendTo(pageViewEventPerMemberStream); } } Unified for Batch & Stream Configuration for Stream Input (Kafka): systems.kafka.samza.factory = org.apache.samza.system.KafkaSystemFactory streams.PageViewEvent.samza.system = kafka streams.PageViewEvent.samza.physical.name = PageViewEvent
  • 49.
    public class CountByDeviceOSApplicationimplements StreamApplication { @Override public void init(StreamGraph graph, Config config) { Supplier<Integer> initialValue = () -> 0; MessageStream<PageViewEvent> pageViewEvents = graph.getInputStream("pageViewEvent", (k, m) -> (PageViewEvent) m); OutputStream<String, MyStreamOutput, MyStreamOutput> pageViewEventPerMemberStream = graph .getOutputStream("pageViewCountPerDevice", m -> m.memberId, m -> m); pageViewEvents .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow( m -> m.memberId, Duration.ofMinutes(5), initialValue, (m, c) -> c + 1)) .map(MyStreamOutput::new) .sendTo(pageViewEventPerMemberStream); } } Unified for Batch & Stream Configuration for Stream Input (Kafka): systems.kafka.samza.factory = org.apache.samza.system.KafkaSystemFactory streams.PageViewEvent.samza.system = kafka streams.PageViewEvent.samza.physical.name = PageViewEvent Configuration for Batch Input (HDFS): systems.hdfs.samza.factory = org.apache.samza.system.HdfsSystemFactory streams.PageViewEvent.samza.system = hdfs streams.PageViewEvent.samza.physical.name = hdfs:/user/nramesh/PageViewEvent
  • 50.
    public class CountByDeviceOSApplicationimplements StreamApplication { @Override public void init(StreamGraph graph, Config config) { Supplier<Integer> initialValue = () -> 0; MessageStream<PageViewEvent> pageViewEvents = graph.getInputStream("pageViewEvent", (k, m) -> (PageViewEvent) m); OutputStream<String, MyStreamOutput, MyStreamOutput> pageViewEventPerMemberStream = graph .getOutputStream("pageViewCountPerDevice", m -> m.memberId, m -> m); pageViewEvents .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow( m -> m.memberId, Duration.ofMinutes(5), initialValue, (m, c) -> c + 1)) .map(MyStreamOutput::new) .sendTo(pageViewEventPerMemberStream); } } Unified for Batch & Stream Configuration for Stream Input (Kafka): systems.kafka.samza.factory = org.apache.samza.system.KafkaSystemFactory streams.PageViewEvent.samza.system = kafka streams.PageViewEvent.samza.physical.name = PageViewEvent Configuration for Batch Input (HDFS): systems.hdfs.samza.factory = org.apache.samza.system.HdfsSystemFactory streams.PageViewEvent.samza.system = hdfs streams.PageViewEvent.samza.physical.name = hdfs:/user/nramesh/PageViewEvent Only Config Change!
  • 51.
    High-level API -Visualization for DAG SAMZA Visualizer A visualization of application samza-count-by-device-i001, which consists of 1 job(s), 1 input stream(s), and 1 output stream(s).
  • 52.
  • 53.
    Agenda ● Data Processingat LinkedIn ● Data Pipelines in Batch & Stream ● Overview of Apache Samza ● Convergence of Pipelines with Apache Samza ○ Support for Batch Data ○ Unified Data Processing API ○ Flexible Deployment Model
  • 54.
    Coordination Model • Coordinationlayer is pluggable in Samza • Samza master / leader – Distributes tasks to processor JVMs – On processor failure, it re-distributes • Available Coordination Mechanisms – Apache Yarn • ApplicationMaster is the leader – Apache Zookeeper • One of the processors is the leader and co-ordinates via Zookeeper – Microsoft Azure • One of the processors is the leader and co-ordinates via Azure’s Blob/Tables Storage
  • 55.
    Embedding Processor withinApplication - An instance of the processor is embedded within user’s application - LocalApplicationRunner helps launch the processor within the application public static void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); CountByDeviceOSApplication app = new CountByDeviceOSApplication(); runner.run(app); runner.waitForFinish(); }
  • 56.
    Pluggable Coordination Config publicstatic void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); CountByDeviceOSApplication app = new CountByDeviceOSApplication(); runner.run(app); runner.waitForFinish(); } Configs with Zk-based coordination job.coordinator.factory = org.apache.samza.zk.ZkJobCoordinatorFactory job.coordinator.zk.connect = foobar:2181/samza
  • 57.
    Pluggable Coordination Config publicstatic void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); CountByDeviceOSApplication app = new CountByDeviceOSApplication(); runner.run(app); runner.waitForFinish(); } Configs with Azure-based coordination: job.coordinator.factory = org.apache.samza.azure.AzureJobCoordinatorFactory job.coordinator.azure.connect = http://foobar:29892/storage/ Configs with Zk-based coordination job.coordinator.factory = org.apache.samza.zk.ZkJobCoordinatorFactory job.coordinator.zk.connect = foobar:2181/samza
  • 58.
    Pluggable Coordination Config publicstatic void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); CountByDeviceOSApplication app = new CountByDeviceOSApplication(); runner.run(app); runner.waitForFinish(); } Only Config Change! Configs with Azure-based coordination: job.coordinator.factory = org.apache.samza.azure.AzureJobCoordinatorFactory job.coordinator.azure.connect = http://foobar:29892/storage/ Configs with Zk-based coordination job.coordinator.factory = org.apache.samza.zk.ZkJobCoordinatorFactory job.coordinator.zk.connect = foobar:2181/samza
  • 59.
    Deploying Samza ina Managed Cluster (Yarn) app.class = MyStreamApplication RemoteAppplicationRunner: main() RM NM LocalApplicationRunner StreamProcessor JobCoordinator NM NM LocalApplicationRunner StreamProcessor Client Submits JAR run-jc.sh run-app.sh run-local-app.sh run-local-app.sh
  • 60.
    Flexible Deployment Models Samzaas a Library - Run embedded stream processing in user program - Use Zookeeper for partition distribution among tasks and liveness of processors - Seamlessly scale by spinning a new processor instance Samza as a Service - Run stream processing as a managed program in a cluster (eg. Yarn) - Works with the cluster manager (Eg. AM/RM) for partition distribution among tasks and liveness of processors - Better for resource sharing in a multi- tenant environment
  • 61.
    Conclusion ● Easily ComposableArchitecture allows varied data source consumption ● Write Once, Run Anywhere paradigm ○ Unified API - application logic to be written only once ○ Pluggable Coordination Model - allows application deployment across different execution environment
  • 62.
    Future Work ● SupportSQL on Streams with Samza ● Table Abstraction in Samza ● Event-time processing ● Samza runner for Apache Beam Contributions are welcome! ● Contributor’s Corner - http://samza.apache.org/contribute/contributors-corner.html ● Ask any question - dev@samza.apache.org ● Follow or tweet us @apachesamza
  • 63.
  • 64.
  • 65.
    Lambda-less Architecture withSamza Profile Updates Kafka stream Standardization Normalized Profile Updates Kafka stream Member Profiles
  • 66.
    Lambda-less Architecture withSamza Profile Updates Kafka stream Standardization Normalized Profile Updates Kafka stream Member Profiles update the standardization job
  • 67.
    Lambda-less Architecture withSamza Profile Updates Kafka stream Standardization Normalized Profile Updates Kafka stream Member Profiles update the standardization job DB Snapshot Standardization Merge & Store Results
  • 68.
    Lambda-less Architecture withSamza Profile Updates Kafka stream Standardization Normalized Profile Updates Kafka stream Member Profiles DB Snapshot Standardization Merge & Store Results Stream Processing Batch Processing
  • 69.