Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Beam me up, Samza!

765 views

Published on

The slides for Stream Processing Meetup (7/19/2018)(https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/251481797/).

This presentation introduces the newly-developed Samza Runner for Apache Beam. You will see the capability of the Samza Runner and how it supports key Beam features. You will also see a few use cases and our future roadmap.

Published in: Engineering
  • Be the first to comment

Beam me up, Samza!

  1. 1. Beam me up, Samza! How we built a Samza Runner for Apache Beam Xinyu Liu Apache Samza PMC Software Engineer@LinkedIn
  2. 2. Agenda ● Apache Beam ● Samza Runner ● Use Cases ● Future Work
  3. 3. Apache Beam Overview Apache Beam is an advanced unified programming model designed to provide efficient and portable data processing pipelines. ● Unified - Single programming model for both batch and streaming ● Advanced - Strong consistency via event-time, i.e. windowing, triggering, late arrival handling, accumulation, etc ● Portable - Execute pipelines of multiple programming language SDKs, including Java, Python and Go ● Efficient - Write and share SDKs, IO connectors, and transformation libraries https://beam.apache.org/
  4. 4. Beam Model ● A Pipeline encapsulates your entire data processing task, from start to finish ● IO is the end points for data input and output ● A PCollection represents an immutable distributed data set that your Beam pipeline operates on ● A PTransform represents a data processing operation, or a step, in your pipeline IO.read IO.write PTransform IO.read IO.write PCollection Pipeline
  5. 5. Beam Model Example public static void main(String[] args) { // Create Pipeline PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create(); Pipeline pipeline = Pipeline.create(options); // Read from KafkaIO PCollection<KafkaRecord<String, PageViewEvent>> input = pipeline .apply(KafkaIO.<String, PageViewEvent>read() .withTopic("PageViewEvent") .withTimestampFn(kv -> new Instant(kv.getValue().timestamp)) .updateConsumerProperties(kafkaConfig) .withKeyDeserializer(StringDeserializer.class) .withValueDeserializer(AvroDeserializer.class)); // Apply PTransforms PCollection<KV<String, Long>> counts = input .apply(MapElements .into(TypeDescriptor.of(PageViewEvent.class)) .via(r -> r.getKV().getValue())) .apply(Window .<PageViewEvent>into(FixedWindows.of(Duration.standardMinutes(1)))) .apply(WithKeys .of((PageViewEvent pv) -> pv.pageKey) .withKeyType(TypeDescriptors.strings())) .apply(Count.perKey()); } KafkaIO.Read (KafkaUnboundedSource) MapElements PCollection<KafkaRecord> Window WithKeys Count.perKey PCollection<PageViewEvent> PCollection<PageViewEvent> PCollection<KV<String, PageViewEvent>> PCollection<KV<String, Long>>
  6. 6. Beam Event Time 12:00 12:01 12:02 12:03 12:02 12:03 12:04 2 3 4 5 6 7 8 1 1-min fixed window using processing time 12:01 ProcessingTime Event time
  7. 7. Beam Event Time 12:00 12:01 12:02 12:03 12:02 12:03 12:04 1 2 1-min fixed window using event time - Watermark: a timestamp that all events before that have arrived. - Data that arrives with a timestamp after the watermark is considered late data. - Example using simple watermark of event timestamp 12:01 watermark ProcessingTime Event time
  8. 8. Beam Event Time 12:00 12:01 12:02 12:03 12:02 12:03 12:04 1 2 3 4 5 6 12:01 1-min fixed window using event time - Watermark: a timestamp that all events before that have arrived. - Data that arrives with a timestamp after the watermark is considered late data. - Example using simple watermark of event timestamp watermark ProcessingTime Event time
  9. 9. Beam Event Time 12:00 12:01 12:02 12:03 12:02 12:03 12:04 1 2 3 4 5 6 7 8 12:01 1-min fixed window using event time - Watermark: a timestamp that all events before that have arrived. - Data that arrives with a timestamp after the watermark is considered late data. - Example using simple watermark of event timestamp watermark lateProcessingTime Event time
  10. 10. Beam Windowing Windowing divides data into event-time-based finite chunks. Often required when doing aggregations over unbounded data. Fixed Sliding 1 2 3 54 Sessions 2 431 Key 2 Key 1 Key 3 Time 2 3 4
  11. 11. Beam Stateful Processing 12:00 12:01 12:02 12:03 12:02 12:03 12:04 12:01 (news) (msg) (msg) (jobs) (msg) (network) (news) 12:00-12:01 12:01-12:02 12:02-12:03 news 1 0 1 msg 0 3 0 network 0 0 1 jobs 0 1 0 ● Beam provides several state abstractions, e.g. ValueState, BagState, MapState, CombineState ● State is on a per-key-and-window basis State for counting PageKey: ProcessingTime Event time
  12. 12. Beam IO In-progress:
  13. 13. Agenda ● Apache Beam Overview ● Samza Runner ● Use Cases ● Future Work
  14. 14. The Goal of Samza Runner Bring the easy-to-use, but powerful, model of Beam to Samza users for state-of-art stream and batch data processing, with portability across a variety of programming languages.
  15. 15. Samza Overview ● The runner combines the large-scale stream processing capabilities of Samza with the the advanced programming model of Beam ● First class support for local state (with RocksDB store) ● Fault-tolerance with support for incremental checkpointing of state instead of full snapshots ● A fully asynchronous processing engine that makes remote calls efficient ● Flexible deployment models, e.g. Yarn and standalone with Zookeeper
  16. 16. Samza Runner: Capability Matrix
  17. 17. How Samza Runner Works? ● A Beam runner translates the Beam API into its native API + runtime logic, and executed it in a distributed data processing system. ● Samza Runner translates Beam API into Samza high-level API and execute the logic in a distributed manner, e.g. Yarn, Standalone. ● Samza runner contains the logic to support Beam features - Beam IO - Event time/Watermark - GroupByKey - Keyed State - Triggering Timers - Side Input
  18. 18. BoundedSourceSystem UnboundedSourceSystem Unbounded/Bounded IO ● UnboundedSourceSystem adapts any Unbounded IO.Read into a Samza SystemConsumer. It will 1) split the sources according to the parallelism needed; 2) generate IncomingMessageEnvelopes of either event or watermark ● BoundedSourceSystem adapts any Bounded IO.Read into a Samza SystemConsumer ● Direct translation is also supported for Samza native data connectors, e.g. translating KafkaIO.Read directly into KafkaSystemConsumer KafkaIO.Read (KafkaUnboundedSource) TextIO.Read (TextSource) Samza StreamProcessor Events/Watermarks Events/End-of-stream
  19. 19. Watermark ● Watermark is injected at a fixed interval from unbounded sources ● Watermarks are propagated through each downstream operators and aggregated using the following logic: InputWatermak(op) = max (CurrentInputWatermark(op), min(OutputWatermark(op') | op' is upstream of op)) OutputWatermark(op) = max (CurrentOutputWatermark(op), InputWatermark(op))
  20. 20. Watermark Example KafkaIO.Read KafkaIO.Read FlatMap Filter Map Merge PartitionBy 0 1 2 3 FlatMap Task Watermark 0 8 1 6 2 10 3 3 -> 7 7 7 7 7 9 9 7 7 6 6 6 6 IntermediateMessage => [MessageType MessageData] MessageType => byte MessageData => byte[] MessageData => [UserMessage/ControlMessage] ControlMessage => Type => int TaskName => String 1. WatermarkMessage => Timestamp => long 2. EndOfStreamMessage Aggregate
  21. 21. GroupByKey GroupByKey ● Automatically inserting partitionBy before reduce ● The intermediate aggregation results are stored in Samza key-value stores (RocksDb by default) ● The output is triggered by watermarks by default KafkaIO.Read partitionBy FlatMap Run ReduceFn State KV<key, value>
  22. 22. State Support ● Beam states are provided by SamzaStoreStateInternals ● The key for each state cell is (element key, window id, address) ● Samza also provides an readIterator() interface for large states that won’t fit in memory ValueState BagState SetState MapState CombingState WatermarkState SamzaStoreStateInternals RocksDb
  23. 23. Timer Support ● Beam timers are provided by SamzaTimerInternalsFactory ● Support both event-time and processing-time timers ● Event-time timers are managed using a sorted set ordered by timestamp ● Processing-time timers are managed by Samza TimerRegistry via TimerFunction API ● All timers are keyed by TimerKey (id, namespace, element key) SamzaTimerInternals Event-time Timers GroupByKey k1 timer1 k2 timer2 k3 timer3 k4 timer4 Processing-time Timers Key Timer k1 timer1 k2 timer2 k3 timer3 setTimer watermark register Samza SystemTimerScheduler
  24. 24. View/Side Input ● Beam views: SingletonView, IterableView, ListView, MapView, MultimapView ● Beam views are materialized into a physical stream and broadcast to all tasks using Samza broadcast operator ● ParDo will consume the broadcasted view as side input KafkaIO.Read 0 1 2 3 ParDo ParDo ParDo ParDo TextIO.Read ParDo Combine. GloballyAsSingletonView 0 broadcast Broadcast Stream Side input Main input
  25. 25. Deployment Local (single JVM) ●Default mode: No config required ●LocalApplicationRunner ●PassthroughJobCoordinator ●All tasks grouped into one container Yarn ●RemoteApplicationRunner ●YarnJobFactory ●Configure containers using job.container.count N M N M N M N M N M N M RM Yarn Cluster JVM Process Standalone (zookeeper) ●LocalApplicationRunner ●ZkJobCoordinator ●Configure zk connection job.coordinator.zk.connect StreamProcessor Samza Contai ner Job Coordi nator StreamProcessor Samza Contai ner Job Coordi nator StreamProcessor Samza Contai ner Job Coordi nator StreamProcessor Samza Contai ner Job Coordi nator Zookeeper
  26. 26. Agenda ● Apache Beam Overview ● Samza Runner ● Use Cases ● Future Work
  27. 27. Use Case 1: Fixed-window Join to Track Location Onboard location transmitt er Radar Monitor WithKey (key by ID) WithKey (key by ID) FixedWindow (10 min) FixedWindow (10 min) CoGroupByKey (join) Location Info DB Suppose you own a Star Trek fleet, and you want to track the location of your Starships. The location data are gathered through Starship on-board transmitters as well as your radar monitors. Now let’s track their location in event time of a 10-min window. (T1, Enterprise, SF, 1) (R1, Enterprise SV, 9) Enterprise, (SF, 1) Enterprise, (SV, 9) Enterprise, (SF, 1) Window(0 : 9) Enterprise, (SV, 9) Window(0 : 9) ParDo KafkaIO.Write DbIO.Write Enterprise, (SF, 1), (SV, 9) Window(0 : 9) Enterprise, SV Window(0 : 9) Enterprise, SV Enterprise, SV
  28. 28. Use Case 2: Session-window Join to Gather Activities Suppose we are heading out to Disneyland. We would like to know the activity count for each person. Here we use session window join to gather the activities done per person. Ticket Purchase Event Membership Purchase Event Activity Event CoGroupByKey SessionWindow (4 hour) SessionWindow (4 hour) SessionWindow (4 hour) Join by id Xinyu: G00001 Boris: M00001 G00001: Space Mountain M00001: Harry Potter M00001: Small World Count.perKey Xinyu: 1 Boris: 2
  29. 29. Use Case 3: Sliding-window Aggr. for Feature Generation Calculate the features of count, top N and sum for particular key for PageView events using a 1-day sliding window with 1-min update interval. PageView Event SlidingWindow (1day, every min) Count.perKey Top.largestPerKey(n) Filtter.by Sum.globally Schema pageViewSchema = RowSqlTypes .builder() .withVarcharField("pageKey") .withTimestampField("timestamp") .build(); PCollection<Row> pageViewsRows = pageViews .apply(MapElements .into(TypeDescriptor.of(Row.class)) .via((PageViewEvent pv) -> Row.withSchema(pageViewSchema) .addValues(pv.pageKey.toString(), new DateTime(pv.time)).build())) .setCoder(pageViewSchema.getRowCoder()); PCollection<KV<String, Long>> counts = pageViewsRows .apply(BeamSql.query( "SELECT COUNT(*) AS `count` FROM pageView " + "GROUP BY pageKey, " + "HOP(timestamp, INTERVAL '1' MINUTE, INTERVAL '1' DAY)")); Alternatively, using SQL:
  30. 30. Agenda ● Apache Beam Overview ● Samza Runner ● Use Cases ● Future Work
  31. 31. Future Work ● Python! ● Async Support ● Table API # A sample word count p =Pipeline(options=pipeline_options) # Read the text file[pattern] into a PCollection. lines = p | 'read' >> ReadFromText(known_args.input) # Count the occurrences of each word. counts = (lines | 'split' >> (ParDo(WordExtractingDoFn()) .with_output_types(unicode)) | 'pair_with_one' >>Map(lambda x: (x, 1)) | 'group' >> GroupByKey() | 'count' >> Map(lambda (word, ones): (word, sum(ones)))) # Format the counts into a PCollection of strings. output = counts | 'format' >>Map(lambda (word, c): '%s: %s' % (word, c)) # Write the output using a "Write" transform that has side effects. # pylint: disable=expression-not-assigned output | 'write' >> WriteToText(known_args.output) result = p.run() result.wait_until_finish() // Use CompletionStage for asynchronous processing input.apply(ParDo.of( new DoFn<InputT, OutputT>() { @ProcessElement public void process (@Element CompletionStage<InputT> element, ...) { element.thenApply(...) } } )); // PTable is the Table abstraction PTable<KV<String, User>> userTable = pipeline.apply( EspressoTable.readWrite() .withDb("dbname") .withTable("user")); pageView .apply(TableParDo.of( new DoFn<KV<String, PageViewEvent>, String>() { @ProcessElement public void processElement(ProcessContext c, @TableContext.Inject TableContext tc) { String id = c.element().getKey(); //table lookup Table<String, User> users = tc.getTable(userTable); User user = settings.get(id); c.output(id + “:” + user.getName().toString()); } }) .withTables(userTable)); // Convenient helper class to do the same thing PCollection<String> result = PCollectionTableJoin .of(pageView, userTable) .into(TypeDescriptors.strings()) .via((pv, user) -> pv.getKey() + “:” + user.getName().toString());
  32. 32. Thank you! And Special Thanks to Our Early Adopters: Yingkai Hu, Froila Dsouza, Zhongen Tao, Nithin Reddy, Bruce Su https://beam.apache.org/documentation/runners/samza/

×