![29
WordCount SummingBird
def wordCount[P <: Platform[P]]
(source: Producer[P, String], store: P#Store[String, Long]) =
source.flatMap { sentence =>
toWords(sentence).map(_ -> 1L)
}.sumByKey(store)
val stormTopology = Storm.remote(“stormName”).plan(wordCount)
val hadoopJob = Scalding(“scaldingName”).plan(wordCount)
Click to enter confidentiality information](https://image.slidesharecdn.com/june111100amclouderashapira-150629234916-lva1-app6891/75/Have-your-Cake-and-Eat-it-Too-Architecture-for-Batch-and-Real-time-processing-29-2048.jpg)
![35
Spark Example
©2014 Cloudera, Inc. All rights reserved.
1. val conf = new SparkConf().setMaster("local[2]”)
2. val sc = new SparkContext(conf)
3. val lines = sc.textFile(path, 2)
4. val words = lines.flatMap(_.split(" "))
5. val pairs = words.map(word => (word, 1))
6. val wordCounts = pairs.reduceByKey(_ + _)
7. wordCounts.print()](https://image.slidesharecdn.com/june111100amclouderashapira-150629234916-lva1-app6891/75/Have-your-Cake-and-Eat-it-Too-Architecture-for-Batch-and-Real-time-processing-35-2048.jpg)
![36
Spark Streaming Example
©2014 Cloudera, Inc. All rights reserved.
1. val conf = new SparkConf().setMaster("local[2]”)
2. val ssc = new StreamingContext(conf, Seconds(1))
3. val lines = ssc.socketTextStream("localhost", 9999)
4. val words = lines.flatMap(_.split(" "))
5. val pairs = words.map(word => (word, 1))
6. val wordCounts = pairs.reduceByKey(_ + _)
7. wordCounts.print()
8. ssc.start()](https://image.slidesharecdn.com/june111100amclouderashapira-150629234916-lva1-app6891/75/Have-your-Cake-and-Eat-it-Too-Architecture-for-Batch-and-Real-time-processing-36-2048.jpg)
Uploaded byDataWorks Summit
Have your Cake and Eat it Too - Architecture for Batch and Real-time processing
This document discusses architectures for processing both streaming and batch data. It begins by explaining why both streaming and batch processing are needed. It then discusses challenges with maintaining separate streaming and batch applications that do the same work. Various architectures and technologies are presented as solutions, including the Kappa architecture, Lambda architecture, SummingBird, Apache Spark, Apache Flink, and bringing your own framework. Specific examples are provided for how to implement word count in many of these technologies for both batch and streaming data.
More Related Content
Similar to Have your Cake and Eat it Too - Architecture for Batch and Real-time processing
More from DataWorks Summit
![[BDD 2025 - Mobile Development] Mobile Engineer and Software Engineer: Are we...](https://cdn.slidesharecdn.com/ss_thumbnails/md-mobileengineerandsoftwareengineerarewestillrelevantsidiqpermana-251127010650-55224ef1-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Mobile Development] Crafting Immersive UI with E2E and AGSL Shade...](https://cdn.slidesharecdn.com/ss_thumbnails/md-craftingimmersiveuiwithe2eandagslshaderveronicaputrianggraini-251124030840-0c677f44-thumbnail.jpg?width=640&height=640&fit=bounds)
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
Have your Cake and Eat it Too - Architecture for Batch and Real-time processing
- 1. Have Your Cake and EatIt Too Architectures for Batch and Stream Processing Speaker name // Speaker title
- 2. 2 Stuff We’ll TalkAbout • Why do we need both streams and batches • Why is it a problem? • Stream-Only Patterns (i.e. Kappa Architecture) • Lambda-Architecture Technologies – SummingBird – Apache Spark – Apache Flink – Bring-your-own-framework
- 3. 3©2014 Cloudera, Inc.All rights reserved. • 15 years of moving data • Formerly consultant • Now Cloudera Engineer: – Sqoop Committer – Kafka – Flume • @gwenshap About Me
- 4. 4 Why Streaming and Batch ©2014Cloudera, Inc. All rights reserved.
- 5. 5 Batch Processing • Storedata somewhere • Read large chunks of data • Do something with data • Sometimes store results
- 6. 6 Batch Examples • Analytics •ETL / ELT • Training machine learning models • Recommendations Click to enter confidentiality information
- 7. 7 Stream Processing • Listento incoming events • Do something with each event • Maybe store events / results Click to enter confidentiality information
- 8. 8 Stream Processing Examples •Anomaly detection, alerts • Monitoring, SLAs • Operational intelligence • Analytics, dashboards • ETL Click to enter confidentiality information
- 9. 9 Streaming & Batch Clickto enter confidentiality information Alerts Monitoring, SLAs Operational Intelligence Risk Analysis Anomaly detection Analytics ETL
- 10. 10 Four Categories • StreamsOnly • Batch Only • Can be done in both • Must be done in both Click to enter confidentiality information ETL Some Analytics
- 11. 11 ETL Most Stream Processingprojects I see involve few simple transformations. • Currency conversion • JSON to Avro • Field extraction • Joining a stream to a static data set • Aggregate on window • Identifying change in trend • Document indexing Click to enter confidentiality information
- 12. 12 Batch || Streaming •Efficient: – Lower CPU utilization – Better network and disk throughput – Fewer locks and waits • Easier administration • Easier integration with RDBMS • Existing expertise • Existing tools • Real-time information Click to enter confidentiality information
- 13. 13 The Problem ©2014 Cloudera,Inc. All rights reserved.
- 14. 14 We Like • Efficiency •Scalability • Fault Tolerance • Recovery from errors • Experimenting with different approaches • Debuggers • Cookies Click to enter confidentiality information
- 15. 15 But… We don’t like Maintainingtwo applications That do the same thing Click to enter confidentiality information
- 16. 16 Do we reallyneed to maintain same app twice? Yes, because: • We are not sure about requirements • We sometimes need to re-process with very high efficiency Not really: • Different apps for batch and streaming • Can re-process with streams • Can error-correct with streams • Can maintain one code-base for batches and streams Click to enter confidentiality information
- 17. 17 Stream-Only Patterns (Kappa Architecture) Click to enterconfidentiality information
- 18. 18 DWH Example Click toenter confidentiality information OLTP DB Sensors, Logs DWH Fact Table (Partitioned) Real Time Fact Tables Dimensio n Dimensio n Dimensio n Views Aggregat es App 1: Stream processing App 2: Occasional load
- 19. 19 We need tofix older data Click to enter confidentiality information 0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 Streaming App v1 Streaming App v2 Real-Time Table Replacement Partition Partitioned Fact Table
- 20. 20 We need tofix older data Click to enter confidentiality information 0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 Streaming App v1 Streaming App v2 Real-Time Table Replacement Partition Partitioned Fact Table
- 21. 21 We need tofix older data Click to enter confidentiality information 0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 Streaming App v2 Real-Time Table
- 22. 22 Lambda- Architecture Technologies Click to enterconfidentiality information
- 23. 23 WordCount in Scala source.flatMap(line=> line.split(" ")) .map(word => (word, 1)) .reduceByKey(_+_) .print()
- 24.
- 25. 25 MapReduce was greatbecause… Very simple abstraction: - Map - Shuffle - Reduce - Type-safe And it has simpler abstractions on top.
- 26. 26 SummingBird • Multi-stage MapReduce •Run on Hadoop, Spark, Storm • Very easy to combine batch and streaming results Click to enter confidentiality information
- 27. 27 API • Platform –Storm, Scalding, Spark… • Producer.source(Platform) <- get data • Producer – collection of events • Transformations – map, filter, merge, leftJoin (lookup) • Output – write(sink), sumByKey(store) • Store – contains aggregate for each key, and reduce operation Click to enter confidentiality information
- 28. 28 Associative Reduce Click toenter confidentiality information
- 29. 29 WordCount SummingBird def wordCount[P<: Platform[P]] (source: Producer[P, String], store: P#Store[String, Long]) = source.flatMap { sentence => toWords(sentence).map(_ -> 1L) }.sumByKey(store) val stormTopology = Storm.remote(“stormName”).plan(wordCount) val hadoopJob = Scalding(“scaldingName”).plan(wordCount) Click to enter confidentiality information
- 30.
- 31. 31 First, there wasthe RDD • Spark is its own execution engine • With high-level API • RDDs are sharded collections • Can be mapped, reduced, grouped, filtered, etc
- 32. 32 DStream DStream DStream Spark Streaming Confidentiality InformationGoes Here Single Pass Source Receiver RDD Source Receiver RDD RDD Filter Count Print Source Receiver RDD RDD RDD Single Pass Filter Count Print Pre-first Batch First Batch Second Batch
- 33. 33 DStream DStream DStreamSpark Streaming Confidentiality InformationGoes Here Single Pass Source Receiver RDD Source Receiver RDD RDD Filter Count Print Source Receiver RDD RDD RDD Single Pass Filter Count Pre-first Batch First Batch Second Batch Stateful RDD 1 Print Stateful RDD 2 Stateful RDD 1
- 34. 34 Compared to SummingBird Differences: •Micro-batches • Completely new execution model • Real joins • Reduce is not limited to Monads • SparkStreaming has Richer API • Summingbird can aggregate batch and stream to one dataset • SparkStreaming runs in debugger Similarities: • Almost same code will run in batch and streams • Use of Scala • Use of functional programing concepts Click to enter confidentiality information
- 35. 35 Spark Example ©2014 Cloudera,Inc. All rights reserved. 1. val conf = new SparkConf().setMaster("local[2]”) 2. val sc = new SparkContext(conf) 3. val lines = sc.textFile(path, 2) 4. val words = lines.flatMap(_.split(" ")) 5. val pairs = words.map(word => (word, 1)) 6. val wordCounts = pairs.reduceByKey(_ + _) 7. wordCounts.print()
- 36. 36 Spark Streaming Example ©2014Cloudera, Inc. All rights reserved. 1. val conf = new SparkConf().setMaster("local[2]”) 2. val ssc = new StreamingContext(conf, Seconds(1)) 3. val lines = ssc.socketTextStream("localhost", 9999) 4. val words = lines.flatMap(_.split(" ")) 5. val pairs = words.map(word => (word, 1)) 6. val wordCounts = pairs.reduceByKey(_ + _) 7. wordCounts.print() 8. ssc.start()
- 37.
- 38. 38 Execution Model You don’twant to know.
- 39. 39 Flink vs SparkStreaming Differences: •Flink is event-by-event streaming, events go through pipeline. • SparkStreaming has good integration with Hbase as state store • “checkpoint barriers” • Optimization based on strong typing • Flink is newer than SparkStreaming, there is less production experience Similarities: • Very similar APIs • Built-in stream-specific operators (windows) • Exactly once guarantees through checkpoints of offsets and state (Flink is limited to small state for now)
- 40. 40 WordCount Batch val env= ExecutionEnvironment.getExecutionEnvironment val text = getTextDataSet(env) val counts = text.flatMap { _.toLowerCase.split("W+") filter { _.nonEmpty } } .map { (_, 1) } .groupBy(0) .sum(1) counts.print() env.execute(“Wordcount Example”)
- 41. 41 WordCount Streaming val env= ExecutionEnvironment.getExecutionEnvironment val text = env.socketTextStream(host, port) val counts = text.flatMap { _.toLowerCase.split("W+") filter { _.nonEmpty } } .map { (_, 1) } .groupBy(0) .sum(1) counts.print() env.execute(“Wordcount Example”)
- 42.
- 43. 43 If the requirementsare simple…
- 44. 44 How difficult itis to parallelize transformations? Simple transformations Are simple
- 45. 45 Just add Kafka Kafkais a reliable data source You can read Batches Microbatches Streams Also allows for re-partitioning Click to enter confidentiality information
- 46. 46 Cluster management • Managingcluster resources used to be difficult • Now: – YARN – Mesos – Docker – Kubernetes Click to enter confidentiality information
- 47. 47 So your appshould… • Allocate resources and track tasks with YARN / Mesos • Read from Kafka (however often you want) • Do simple transformations • Write to Kafka / Hbase • How difficult can it possibly be? Click to enter confidentiality information
- 48. 48 Parting Thoughts Click toenter confidentiality information
- 49. 49 Good engineering lessons •DRY – do you really need same code twice? • Error correction is critical • Reliability guarantees are critical • Debuggers are really nice • Latency / Throughput trade-offs • Use existing expertise • Stream processing is about patterns
- 50.