Download as POTX, PPTX
![● Embedded application code example
public class WikipediaZkLocalApplication {
/**
* Executes the application using the local application runner.
* It takes two required command line arguments
* config-factory: a fully {@link org.apache.samza.config.factories.PropertiesConfigFactory} class name
* config-path: path to application properties
*
* @param args command line arguments
*/
public static void main(String[] args) {
CommandLine cmdLine = new CommandLine();
OptionSet options = cmdLine.parser().parse(args);
Config config = cmdLine.loadConfig(options);
LocalApplicationRunner runner = new LocalApplicationRunner(config);
WikipediaApplication app = new WikipediaApplication();
runner.run(app);
runner.waitForFinish();
}
}
19](https://image.slidesharecdn.com/nextconsamzapreso-july-final-170825001711/75/Nextcon-samza-preso-july-final-19-2048.jpg)
![● Embedded application code example
public class WikipediaZkLocalApplication {
/**
* Executes the application using the local application runner.
* It takes two required command line arguments
* config-factory: a fully {@link org.apache.samza.config.factories.PropertiesConfigFactory} class name
* config-path: path to application properties
*
* @param args command line arguments
*/
public static void main(String[] args) {
CommandLine cmdLine = new CommandLine();
OptionSet options = cmdLine.parser().parse(args);
Config config = cmdLine.loadConfig(options);
LocalApplicationRunner runner = new LocalApplicationRunner(config);
WikipediaApplication app = new WikipediaApplication();
runner.run(app);
runner.waitForFinish();
}
}
20
job.coordinator.factory=org.apache.samza.zk.ZkJobCoordinatorFactory
job.coordinator.zk.connect=my-zk.server:2191](https://image.slidesharecdn.com/nextconsamzapreso-july-final-170825001711/75/Nextcon-samza-preso-july-final-20-2048.jpg)
Uploaded byYi Pan
Nextcon samza preso july - final
This document discusses Samza, an open source stream processing framework. It provides an overview of Samza's core capabilities including its streaming APIs, flexible deployment options, and convergence of stream and batch processing. The key points are: - Samza allows writing streaming applications using either low-level or new high-level APIs, with the latter supporting pipeline composition and built-in transformations. - Samza applications can be deployed either embedded or in a cluster via YARN, and a new Zookeeper-based coordination model enables flexible deployments. - Samza aims to unify stream and batch processing by allowing streaming applications to read from and write to batch systems like HDFS, without code changes.
More Related Content
Similar to Nextcon samza preso july - final
Nextcon samza preso july - final
- 1. Yi Pan Streams Team@LinkedIn Committer and PMC Chair, Apache Samza 1
- 2. class PageKeyViewsCounterTask implementsStreamTask, InitableTask { public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) { GenericRecord record = ((GenericRecord) envelope.getMsg()); String pageKey = record.get("page-key").toString(); int newCount = pageKeyViews.get(pageKey).incrementAndGet(); collector.send(countStream, pageKey, newCount); } public void init(Config config, TaskContext context) { pageKeyViews = (KeyValueStore<String, Counter>) context.getStore(“myPageKeyViews); } } Task-0 Task-1 Task-2 Deployed via YARN
- 3. Pros ◦ SimpleAPI ◦ Built-in support for states ◦ Leverage YARN for fault-tolerance ◦ High performance (1.2 Mqps / host) Cons ◦ Not easy to write end-to-end processing pipeline in a single program ◦ Deployment is tightly coupled with YARN ◦ No support to run as batch job
- 4. • High-level API •Flexible Deployment Model • Convergence between Batch and Stream Processing 4
- 5. Application logic: CountPageViewEvent for each member in a 5 minute window and send the counts to PageViewEventPerMemberStream Re-partition by memberId window map sendTo PageViewEvent PageViewEventPerMembe rStream 5
- 6. Re-partition window mapsendTo PageViewEvent PageViewEventByMe mberId PageViewEventPerMembe rStream Job-1: PageViewRepartitionTask Job-2: PageViewByMemberIdCounterTask Application in low-level API 6
- 7. • Job-1: Repartitionjob public class PageViewRepartitionTask implements StreamTask { private final SystemStream pageViewByMIDStream = new SystemStream("kafka", "PaveViewEventByMemberId"); @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); collector.send(new OutgoingMessageEnvelope(pageViewByMIDStream, pve.memberId, pve)); } } 7
- 8. • Job-2: Window-basedcounter public class PageViewByMemberIdCounterTask implements InitableTask, StreamTask, WindowableTask { private final SystemStream pageViewCounterStream = new SystemStream("kafka", "PageViewEventPerMemberStream"); private KeyValueStore<String, PageViewPerMemberIdCounterEvent> windowedCounters; private Long windowSize; @Override public void init(Config config, TaskContext context) throws Exception { this.windowedCounters = (KeyValueStore<String, PageViewPerMemberIdCounterEvent>) context.getStore("windowed-counter-store"); this.windowSize = config.getLong("task.window.ms"); } @Override public void window(MessageCollector collector, TaskCoordinator coordinator) throws Exception { getWindowCounterEvent().forEach(counter -> collector.send(new OutgoingMessageEnvelope(pageViewCounterStream, counter.memberId, counter))); } @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); countPageViewEvent(pve); } } 8
- 9. • Job-2: Window-basedcounter public class PageViewByMemberIdCounterTask implements InitableTask, StreamTask, WindowableTask { ... List<PageViewPerMemberIdCounterEvent> getWindowCounterEvent() { List<PageViewPerMemberIdCounterEvent> retList = new ArrayList<>(); Long currentTimestamp = System.currentTimeMillis(); Long cutoffTimestamp = currentTimestamp - this.windowSize; String lowerBound = String.format("%08d-", cutoffTimestamp); String upperBound = String.format("%08d-", currentTimestamp + 1); this.windowedCounters.range(lowerBound, upperBound).forEachRemaining(entry -> retList.add(entry.getValue())); return retList; } void countPageViewEvent(PageViewEvent pve) { String key = String.format("%08d-%s", (pve.timestamp - pve.timestamp % this.windowSize), pve.memberId); PageViewPerMemberIdCounterEvent counter = this.windowedCounters.get(key); if (counter == null) { counter = new PageViewPerMemberIdCounterEvent(pve.memberId, (pve.timestamp - pve.timestamp % this.windowSize), 0); } counter.count ++; this.windowedCounters.put(key, counter); } } 9
- 10. • Samza HighLevel API (NEW) – Ability to express a multi-stage processing pipeline in a single user program – Built-in library to provide high-level stream transformation functions 10
- 11. public class RepartitionAndCounterExampleimplements StreamApplication { @Override public void init(StreamGraph graph, Config config) { Supplier<Integer> initialValue = () -> 0; MessageStream<PageViewEvent> pageViewEvents = graph.getInputStream("pageViewEventStream", (k, m) -> (PageViewEvent) m); OutputStream<String, MyStreamOutput, MyStreamOutput> pageViewEventPerMemberStream = graph .getOutputStream("pageViewEventPerMemberStream", m -> m.memberId, m -> m); pageViewEvents .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow(m -> m.memberId, Duration.ofMinutes(5), initialValue, (m, c) -> c + 1)) .map(MyStreamOutput::new) .sendTo(pageViewEventPerMemberStream); } } Built-in transform functions 11
- 12. • Visualized executionplan Visualization: 12
- 13. • Built-in transformationfunctions in high-level API filter select a subset of messages from the stream map map one input message to an output message flatMap map one input message to 0 or more output messages merge union all inputs into a single output stream partitionBy re-partition the input messages based on a specific field sendTo send the result to an output stream sink send the result to an external system (e.g. external DB) window window aggregation on the input stream join join messages from two input streams stateless functions I/O functions stateful functions 13
- 14. • High-level API •Flexible Deployment Model • Convergence between Batch and Stream Processing 14
- 15. Tight dependencyon YARN Can’t easily port over to non-YARN clusters (e.g. Mesos, Kubernetes, AWS) Can’t directly embed stream processing in other application (eg. a web frontend) 15
- 16. • Flexible deploymentof Samza applications – Samza-as-a-library (NEW) • Run embedded stream processing in a user program • Zookeeper based coordination between multiple instances of user program – Samza in a cluster • Run stream processing as a managed program in a cluster (e.g. SamzaContainer in YARN) • Use the cluster manager (e.g. YARN) to provide deployment, coordination, and resource management 16
- 17. Samza Job iscomposed of a collection of standalone processes ● Full control on ● Application’s life cycle ● Physical resource allocated to Samza processors ● Configuration and initialization StreamProcessor Samza Container Job Coordinator StreamProcessor Samza Container Job Coordinator StreamProcessor Samza Container Job Coordinator... Leader 17
- 18. ● ZooKeeper-based JobCoordinator(stateful use case) ● JobCoordinator uses ZooKeeper for leader election ● Leader will perform partition assignments among all active StreamProcessors ZooKeeper StreamProcessor Samza Container Job Coordinator StreamProcessor Samza Container Job Coordinator StreamProcessor Samza Container Job Coordinator... 18
- 19. ● Embedded applicationcode example public class WikipediaZkLocalApplication { /** * Executes the application using the local application runner. * It takes two required command line arguments * config-factory: a fully {@link org.apache.samza.config.factories.PropertiesConfigFactory} class name * config-path: path to application properties * * @param args command line arguments */ public static void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); WikipediaApplication app = new WikipediaApplication(); runner.run(app); runner.waitForFinish(); } } 19
- 20. ● Embedded applicationcode example public class WikipediaZkLocalApplication { /** * Executes the application using the local application runner. * It takes two required command line arguments * config-factory: a fully {@link org.apache.samza.config.factories.PropertiesConfigFactory} class name * config-path: path to application properties * * @param args command line arguments */ public static void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); WikipediaApplication app = new WikipediaApplication(); runner.run(app); runner.waitForFinish(); } } 20 job.coordinator.factory=org.apache.samza.zk.ZkJobCoordinatorFactory job.coordinator.zk.connect=my-zk.server:2191
- 21. • Embedded applicationlaunch sequence myApp.main() Stream Application Local Application Runner Stream Processor runner.run() streamProcessor.start() n 21
- 22. • Cluster-based applicationlaunch sequence run-app.sh Remote Application Runner JobRunnerjobRunner.run() n main() app.class=my.app.MyStreamApplication Yarn RM run-jc.sh task.execute=run-local-app.sh run-local-app.sh Stream Application myApp.main() Local Application Runner Stream Processor runner.run() streamProcessor.start() n Job Coordinator 22
- 23.
- 24. • High-level API •Flexible Deployment Model • Convergence between Batch and Stream Processing 24
- 25. Application logic: CountPageViewEvent for each member in a 5 minute window and send the counts to PageViewEventPerMemberStream Re-partition by memberId window map sendTo PageViewEvent PageViewEventPerMemb erStream HDFS PageViewEvent: hdfs://mydbsnapshot/PageViewEvent/ PageViewEventPerMemberStream: hdfs://myoutputdb/PageViewEventPerMemberFiles 25
- 26. • No codechange in application streams.pageViewEventStream.system=kafka streams.pageViewEventPerMemberStream.system=kafka streams.pageViewEventStream.system=hdfs streams.pageViewEventStream.physical.name=hdfs://mydbsnapshot/PageViewEvent/ streams.pageViewEventPerMemberStream.system=hdfs streams.pageViewEventPerMemberStream.physical.name=hdfs://myoutputdb/PageViewEventPerMemberFiles old config new config 26
- 27. 27 High-level API Unified Stream& Batch Processing Remote Runner Run in Remote Cluster Cluster-based Yarn, (Mesos) Local Runner Run Locally Embedded ZooKeeper, Standalone APIRUNNERDEPLO YMENT PROCESSO R StreamProcessor Streams Kafka, Kinesis, HDFS ... Local State RocksDb, In-Memory Remote Data Multithreading 27
- 28. Samza runnerfor Apache Beam Event-time processing Support for Exactly-once processing Support partition expansion for stateful application Easy access to Adjunct datasets SQL over Streams 28
- 29.