Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Nextcon samza preso july - final

329 views

Published on

Talk of Samza new API in Cloud + Data Next Conference @Santa Clara '17

Published in: Software
  • Be the first to comment

Nextcon samza preso july - final

  1. 1. Yi Pan Streams Team @LinkedIn Committer and PMC Chair, Apache Samza 1
  2. 2. class PageKeyViewsCounterTask implements StreamTask, InitableTask { public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) { GenericRecord record = ((GenericRecord) envelope.getMsg()); String pageKey = record.get("page-key").toString(); int newCount = pageKeyViews.get(pageKey).incrementAndGet(); collector.send(countStream, pageKey, newCount); } public void init(Config config, TaskContext context) { pageKeyViews = (KeyValueStore<String, Counter>) context.getStore(“myPageKeyViews); } } Task-0 Task-1 Task-2 Deployed via YARN
  3. 3.  Pros ◦ Simple API ◦ Built-in support for states ◦ Leverage YARN for fault-tolerance ◦ High performance (1.2 Mqps / host)  Cons ◦ Not easy to write end-to-end processing pipeline in a single program ◦ Deployment is tightly coupled with YARN ◦ No support to run as batch job
  4. 4. • High-level API • Flexible Deployment Model • Convergence between Batch and Stream Processing 4
  5. 5. Application logic: Count PageViewEvent for each member in a 5 minute window and send the counts to PageViewEventPerMemberStream Re-partition by memberId window map sendTo PageViewEvent PageViewEventPerMembe rStream 5
  6. 6. Re-partition window map sendTo PageViewEvent PageViewEventByMe mberId PageViewEventPerMembe rStream Job-1: PageViewRepartitionTask Job-2: PageViewByMemberIdCounterTask Application in low-level API 6
  7. 7. • Job-1: Repartition job public class PageViewRepartitionTask implements StreamTask { private final SystemStream pageViewByMIDStream = new SystemStream("kafka", "PaveViewEventByMemberId"); @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); collector.send(new OutgoingMessageEnvelope(pageViewByMIDStream, pve.memberId, pve)); } } 7
  8. 8. • Job-2: Window-based counter public class PageViewByMemberIdCounterTask implements InitableTask, StreamTask, WindowableTask { private final SystemStream pageViewCounterStream = new SystemStream("kafka", "PageViewEventPerMemberStream"); private KeyValueStore<String, PageViewPerMemberIdCounterEvent> windowedCounters; private Long windowSize; @Override public void init(Config config, TaskContext context) throws Exception { this.windowedCounters = (KeyValueStore<String, PageViewPerMemberIdCounterEvent>) context.getStore("windowed-counter-store"); this.windowSize = config.getLong("task.window.ms"); } @Override public void window(MessageCollector collector, TaskCoordinator coordinator) throws Exception { getWindowCounterEvent().forEach(counter -> collector.send(new OutgoingMessageEnvelope(pageViewCounterStream, counter.memberId, counter))); } @Override public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception { PageViewEvent pve = (PageViewEvent) envelope.getMessage(); countPageViewEvent(pve); } } 8
  9. 9. • Job-2: Window-based counter public class PageViewByMemberIdCounterTask implements InitableTask, StreamTask, WindowableTask { ... List<PageViewPerMemberIdCounterEvent> getWindowCounterEvent() { List<PageViewPerMemberIdCounterEvent> retList = new ArrayList<>(); Long currentTimestamp = System.currentTimeMillis(); Long cutoffTimestamp = currentTimestamp - this.windowSize; String lowerBound = String.format("%08d-", cutoffTimestamp); String upperBound = String.format("%08d-", currentTimestamp + 1); this.windowedCounters.range(lowerBound, upperBound).forEachRemaining(entry -> retList.add(entry.getValue())); return retList; } void countPageViewEvent(PageViewEvent pve) { String key = String.format("%08d-%s", (pve.timestamp - pve.timestamp % this.windowSize), pve.memberId); PageViewPerMemberIdCounterEvent counter = this.windowedCounters.get(key); if (counter == null) { counter = new PageViewPerMemberIdCounterEvent(pve.memberId, (pve.timestamp - pve.timestamp % this.windowSize), 0); } counter.count ++; this.windowedCounters.put(key, counter); } } 9
  10. 10. • Samza High Level API (NEW) – Ability to express a multi-stage processing pipeline in a single user program – Built-in library to provide high-level stream transformation functions 10
  11. 11. public class RepartitionAndCounterExample implements StreamApplication { @Override public void init(StreamGraph graph, Config config) { Supplier<Integer> initialValue = () -> 0; MessageStream<PageViewEvent> pageViewEvents = graph.getInputStream("pageViewEventStream", (k, m) -> (PageViewEvent) m); OutputStream<String, MyStreamOutput, MyStreamOutput> pageViewEventPerMemberStream = graph .getOutputStream("pageViewEventPerMemberStream", m -> m.memberId, m -> m); pageViewEvents .partitionBy(m -> m.memberId) .window(Windows.keyedTumblingWindow(m -> m.memberId, Duration.ofMinutes(5), initialValue, (m, c) -> c + 1)) .map(MyStreamOutput::new) .sendTo(pageViewEventPerMemberStream); } } Built-in transform functions 11
  12. 12. • Visualized execution plan Visualization: 12
  13. 13. • Built-in transformation functions in high-level API filter select a subset of messages from the stream map map one input message to an output message flatMap map one input message to 0 or more output messages merge union all inputs into a single output stream partitionBy re-partition the input messages based on a specific field sendTo send the result to an output stream sink send the result to an external system (e.g. external DB) window window aggregation on the input stream join join messages from two input streams stateless functions I/O functions stateful functions 13
  14. 14. • High-level API • Flexible Deployment Model • Convergence between Batch and Stream Processing 14
  15. 15.  Tight dependency on YARN  Can’t easily port over to non-YARN clusters (e.g. Mesos, Kubernetes, AWS)  Can’t directly embed stream processing in other application (eg. a web frontend) 15
  16. 16. • Flexible deployment of Samza applications – Samza-as-a-library (NEW) • Run embedded stream processing in a user program • Zookeeper based coordination between multiple instances of user program – Samza in a cluster • Run stream processing as a managed program in a cluster (e.g. SamzaContainer in YARN) • Use the cluster manager (e.g. YARN) to provide deployment, coordination, and resource management 16
  17. 17. Samza Job is composed of a collection of standalone processes ● Full control on ● Application’s life cycle ● Physical resource allocated to Samza processors ● Configuration and initialization StreamProcessor Samza Container Job Coordinator StreamProcessor Samza Container Job Coordinator StreamProcessor Samza Container Job Coordinator... Leader 17
  18. 18. ● ZooKeeper-based JobCoordinator (stateful use case) ● JobCoordinator uses ZooKeeper for leader election ● Leader will perform partition assignments among all active StreamProcessors ZooKeeper StreamProcessor Samza Container Job Coordinator StreamProcessor Samza Container Job Coordinator StreamProcessor Samza Container Job Coordinator... 18
  19. 19. ● Embedded application code example public class WikipediaZkLocalApplication { /** * Executes the application using the local application runner. * It takes two required command line arguments * config-factory: a fully {@link org.apache.samza.config.factories.PropertiesConfigFactory} class name * config-path: path to application properties * * @param args command line arguments */ public static void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); WikipediaApplication app = new WikipediaApplication(); runner.run(app); runner.waitForFinish(); } } 19
  20. 20. ● Embedded application code example public class WikipediaZkLocalApplication { /** * Executes the application using the local application runner. * It takes two required command line arguments * config-factory: a fully {@link org.apache.samza.config.factories.PropertiesConfigFactory} class name * config-path: path to application properties * * @param args command line arguments */ public static void main(String[] args) { CommandLine cmdLine = new CommandLine(); OptionSet options = cmdLine.parser().parse(args); Config config = cmdLine.loadConfig(options); LocalApplicationRunner runner = new LocalApplicationRunner(config); WikipediaApplication app = new WikipediaApplication(); runner.run(app); runner.waitForFinish(); } } 20 job.coordinator.factory=org.apache.samza.zk.ZkJobCoordinatorFactory job.coordinator.zk.connect=my-zk.server:2191
  21. 21. • Embedded application launch sequence myApp.main() Stream Application Local Application Runner Stream Processor runner.run() streamProcessor.start() n 21
  22. 22. • Cluster-based application launch sequence run-app.sh Remote Application Runner JobRunnerjobRunner.run() n main() app.class=my.app.MyStreamApplication Yarn RM run-jc.sh task.execute=run-local-app.sh run-local-app.sh Stream Application myApp.main() Local Application Runner Stream Processor runner.run() streamProcessor.start() n Job Coordinator 22
  23. 23. 23
  24. 24. • High-level API • Flexible Deployment Model • Convergence between Batch and Stream Processing 24
  25. 25. Application logic: Count PageViewEvent for each member in a 5 minute window and send the counts to PageViewEventPerMemberStream Re-partition by memberId window map sendTo PageViewEvent PageViewEventPerMemb erStream HDFS PageViewEvent: hdfs://mydbsnapshot/PageViewEvent/ PageViewEventPerMemberStream: hdfs://myoutputdb/PageViewEventPerMemberFiles 25
  26. 26. • No code change in application streams.pageViewEventStream.system=kafka streams.pageViewEventPerMemberStream.system=kafka streams.pageViewEventStream.system=hdfs streams.pageViewEventStream.physical.name=hdfs://mydbsnapshot/PageViewEvent/ streams.pageViewEventPerMemberStream.system=hdfs streams.pageViewEventPerMemberStream.physical.name=hdfs://myoutputdb/PageViewEventPerMemberFiles old config new config 26
  27. 27. 27 High-level API Unified Stream & Batch Processing Remote Runner Run in Remote Cluster Cluster-based Yarn, (Mesos) Local Runner Run Locally Embedded ZooKeeper, Standalone APIRUNNERDEPLO YMENT PROCESSO R StreamProcessor Streams Kafka, Kinesis, HDFS ... Local State RocksDb, In-Memory Remote Data Multithreading 27
  28. 28.  Samza runner for Apache Beam  Event-time processing  Support for Exactly-once processing  Support partition expansion for stateful application  Easy access to Adjunct datasets  SQL over Streams 28
  29. 29. Q&A 29

×