Alexey Zinoviev, profile picture
Uploaded byAlexey Zinoviev
PDF, PPTX1,226 views

Kafka pours and Spark resolves

The document discusses the integration of Apache Spark and Kafka for real-time data processing, highlighting key concepts such as Spark Streaming and DStreams. It covers the architecture of Kafka as a distributed messaging system and provides practical examples of using Spark with Kafka, including word count operations and structured streaming. Additionally, it mentions considerations for performance and capabilities of Spark Streaming, leading to the conclusion that a scalable, fault-tolerant real-time pipeline is feasible with these technologies.

Embed presentation

Download as PDF, PPTX
1 / 113
2 / 113
3 / 113
4 / 113
5 / 113
6 / 113
7 / 113
8 / 113
9 / 113
10 / 113
11 / 113
12 / 113
13 / 113
14 / 113
15 / 113
16 / 113
17 / 113
18 / 113
19 / 113
20 / 113
21 / 113
22 / 113
23 / 113
24 / 113
25 / 113
26 / 113
27 / 113
28 / 113
29 / 113
30 / 113
31 / 113
32 / 113
33 / 113
34 / 113
35 / 113
36 / 113
37 / 113
38 / 113
39 / 113
40 / 113
41 / 113
42 / 113
43 / 113
44 / 113
45 / 113
46 / 113
47 / 113
48 / 113
49 / 113
50 / 113
51 / 113
52 / 113
53 / 113
54 / 113
55 / 113
56 / 113
57 / 113
58 / 113
59 / 113
60 / 113
61 / 113
62 / 113
63 / 113
64 / 113
65 / 113
66 / 113
67 / 113
68 / 113
69 / 113
70 / 113
71 / 113
72 / 113
73 / 113
74 / 113
75 / 113
76 / 113
77 / 113
78 / 113
79 / 113
80 / 113
81 / 113
82 / 113
83 / 113
84 / 113
85 / 113
86 / 113
87 / 113
88 / 113
89 / 113
90 / 113
91 / 113
92 / 113
93 / 113
94 / 113
95 / 113
96 / 113
97 / 113
98 / 113
99 / 113
100 / 113
101 / 113
102 / 113
103 / 113
104 / 113
105 / 113
106 / 113
107 / 113
108 / 113
109 / 113
110 / 113
111 / 113
112 / 113
113 / 113
Kafka pours and Spark resolves! Alexey Zinovyev, Java/BigData Trainer in EPAM
About With IT since 2007 With Java since 2009 With Hadoop since 2012 With Spark since 2014 With EPAM since 2015
3Spark Streaming from Zinoviev Alexey Contacts E-mail : Alexey_Zinovyev@epam.com Twitter : @zaleslaw @BigDataRussia Facebook: https://www.facebook.com/zaleslaw vk.com/big_data_russia Big Data Russia vk.com/java_jvm Java & JVM langs
4Spark Streaming from Zinoviev Alexey Spark Family
5Spark Streaming from Zinoviev Alexey Spark Family
6Spark Streaming from Zinoviev Alexey Pre-summary • Before RealTime • Spark + Cassandra • Sending messages with Kafka • DStream Kafka Consumer • Structured Streaming in Spark 2.1 • Kafka Writer in Spark 2.2
7Spark Streaming from Zinoviev Alexey < REAL-TIME
8Spark Streaming from Zinoviev Alexey Modern Java in 2016Big Data in 2014
9Spark Streaming from Zinoviev Alexey Batch jobs produce reports. More and more..
10Spark Streaming from Zinoviev Alexey But customer can wait forever (ok, 1h)
11Spark Streaming from Zinoviev Alexey Big Data in 2017
12Spark Streaming from Zinoviev Alexey Machine Learning EVERYWHERE
13Spark Streaming from Zinoviev Alexey Data Lake in promotional brochure
14Spark Streaming from Zinoviev Alexey Data Lake in production
15Spark Streaming from Zinoviev Alexey Simple Flow in Reporting/BI systems
16Spark Streaming from Zinoviev Alexey Let’s use Spark. It’s fast!
17Spark Streaming from Zinoviev Alexey MapReduce vs Spark
18Spark Streaming from Zinoviev Alexey MapReduce vs Spark
19Spark Streaming from Zinoviev Alexey Simple Flow in Reporting/BI systems with Spark
20Spark Streaming from Zinoviev Alexey Spark handles last year logs with ease
21Spark Streaming from Zinoviev Alexey Where can we store events?
22Spark Streaming from Zinoviev Alexey Let’s use Cassandra to store events!
23Spark Streaming from Zinoviev Alexey Let’s use Cassandra to read events!
24Spark Streaming from Zinoviev Alexey Cassandra CREATE KEYSPACE mySpace WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1 }; USE test; CREATE TABLE logs ( application TEXT, time TIMESTAMP, message TEXT, PRIMARY KEY (application, time));
25Spark Streaming from Zinoviev Alexey Cassandra to Spark val dataSet = sqlContext .read .format("org.apache.spark.sql.cassandra") .options(Map( "table" -> "logs", "keyspace" -> "mySpace" )) .load() dataSet .filter("message = 'Log message'") .show()
26Spark Streaming from Zinoviev Alexey Simple Flow in Pre-Real-Time systems
27Spark Streaming from Zinoviev Alexey Spark cluster over Cassandra Cluster
28Spark Streaming from Zinoviev Alexey More events every second!
29Spark Streaming from Zinoviev Alexey SENDING MESSAGES
30Spark Streaming from Zinoviev Alexey Your Father’s Messaging System
31Spark Streaming from Zinoviev Alexey Your Father’s Messaging System
32Spark Streaming from Zinoviev Alexey Your Father’s Messaging System InitialContext ctx = new InitialContext(); QueueConnectionFactory f = (QueueConnectionFactory)ctx.lookup(“qCFactory"); QueueConnection con = f.createQueueConnection(); con.start();
33Spark Streaming from Zinoviev Alexey KAFKA
34Spark Streaming from Zinoviev Alexey Kafka • messaging system
35Spark Streaming from Zinoviev Alexey Kafka • messaging system • distributed
36Spark Streaming from Zinoviev Alexey Kafka • messaging system • distributed • supports Publish-Subscribe model
37Spark Streaming from Zinoviev Alexey Kafka • messaging system • distributed • supports Publish-Subscribe model • persists messages on disk
38Spark Streaming from Zinoviev Alexey Kafka • messaging system • distributed • supports Publish-Subscribe model • persists messages on disk • replicates within the cluster (integrated with Zookeeper)
39Spark Streaming from Zinoviev Alexey The main benefits of Kafka Scalability with zero down time Zero data loss due to replication
40Spark Streaming from Zinoviev Alexey Kafka Cluster consists of … • brokers (leader or follower) • topics ( >= 1 partition) • partitions • partition offsets • replicas of partition • producers/consumers
41Spark Streaming from Zinoviev Alexey Kafka Components with topic “messages” #1 Producer Thread #1 Producer Thread #2 Producer Thread #3 Topic: Messages Data Data Data Zookeeper
42Spark Streaming from Zinoviev Alexey Kafka Components with topic “messages” #2 Producer Thread #1 Producer Thread #2 Producer Thread #3 Topic: Messages Part #1 Part #2 Data Data Data Broker #1 Zookeeper Part #1 Part #2 Leader Leader
43Spark Streaming from Zinoviev Alexey Kafka Components with topic “messages” #3 Producer Thread #1 Producer Thread #2 Producer Thread #3 Topic: Messages Part #1 Part #2 Data Data Data Broker #1 Broker #2 Zookeeper Part #1 Part #1 Part #2 Part #2
44Spark Streaming from Zinoviev Alexey Why do we need Zookeeper?
45Spark Streaming from Zinoviev Alexey Kafka Demo
46Spark Streaming from Zinoviev Alexey REAL TIME WITH DSTREAMS
47Spark Streaming from Zinoviev Alexey RDD Factory
48Spark Streaming from Zinoviev Alexey From socket to console with DStreams
49Spark Streaming from Zinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
50Spark Streaming from Zinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
51Spark Streaming from Zinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
52Spark Streaming from Zinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
53Spark Streaming from Zinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
54Spark Streaming from Zinoviev Alexey Kafka as a main entry point for Spark
55Spark Streaming from Zinoviev Alexey DStreams Demo
56Spark Streaming from Zinoviev Alexey How to avoid DStreams with RDD-like API?
57Spark Streaming from Zinoviev Alexey SPARK 2.2 DISCUSSION
58Spark Streaming from Zinoviev Alexey Continuous Applications
59Spark Streaming from Zinoviev Alexey Continuous Applications cases • Updating data that will be served in real time • Extract, transform and load (ETL) • Creating a real-time version of an existing batch job • Online machine learning
60Spark Streaming from Zinoviev Alexey The main concept of Structured Streaming You can express your streaming computation the same way you would express a batch computation on static data.
61Spark Streaming from Zinoviev Alexey Batch Spark 2.2 // Read JSON once from S3 logsDF = spark.read.json("s3://logs") // Transform with DataFrame API and save logsDF.select("user", "url", "date") .write.parquet("s3://out")
62Spark Streaming from Zinoviev Alexey Real Time Spark 2.2 // Read JSON continuously from S3 logsDF = spark.readStream.json("s3://logs") // Transform with DataFrame API and save logsDF.select("user", "url", "date") .writeStream.parquet("s3://out") .start()
63Spark Streaming from Zinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
64Spark Streaming from Zinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
65Spark Streaming from Zinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
66Spark Streaming from Zinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count() Don’t forget to start Streaming
67Spark Streaming from Zinoviev Alexey Unlimited Table
68Spark Streaming from Zinoviev Alexey WordCount with Structured Streaming [Complete Mode]
69Spark Streaming from Zinoviev Alexey Kafka -> Structured Streaming -> Console
70Spark Streaming from Zinoviev Alexey Kafka To Console Demo
71Spark Streaming from Zinoviev Alexey OPERATIONS
72Spark Streaming from Zinoviev Alexey You can … • filter • sort • aggregate • join • foreach • explain
73Spark Streaming from Zinoviev Alexey Operators Demo
74Spark Streaming from Zinoviev Alexey How it works?
75Spark Streaming from Zinoviev Alexey Deep Diving in Spark Internals Dataset
76Spark Streaming from Zinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan
77Spark Streaming from Zinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan Optimized Logical Plan
78Spark Streaming from Zinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan Optimized Logical Plan Logical Plan Logical Plan Logical PlanPhysical Plan
79Spark Streaming from Zinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan Optimized Logical Plan Logical Plan Logical Plan Logical PlanPhysical Plan Selected Physical Plan Planner
80Spark Streaming from Zinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan Optimized Logical Plan Logical Plan Logical Plan Logical PlanPhysical Plan Selected Physical Plan
81Spark Streaming from Zinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan Optimized Logical Plan Logical Plan Logical Plan Logical PlanPhysical Plan Selected Physical Plan Incremental #1 Incremental #2 Incremental #3
82Spark Streaming from Zinoviev Alexey Incremental Execution: Planner polls Planner
83Spark Streaming from Zinoviev Alexey Incremental Execution: Planner runs Planner Offset: 1 - 5 Incremental #1 R U N Count: 5
84Spark Streaming from Zinoviev Alexey Incremental Execution: Planner runs #2 Planner Offset: 1 - 5 Incremental #1 Incremental #2 Offset: 6 - 9 R U N Count: 5 Count: 4
85Spark Streaming from Zinoviev Alexey Aggregation with State Planner Offset: 1 - 5 Incremental #1 Incremental #2 Offset: 6 - 9 R U N Count: 5 Count: 5+4=9 5
86Spark Streaming from Zinoviev Alexey DataSet.explain() == Physical Plan == Project [avg(price)#43,carat#45] +- SortMergeJoin [color#21], [color#47] :- Sort [color#21 ASC], false, 0 : +- TungstenExchange hashpartitioning(color#21,200), None : +- Project [avg(price)#43,color#21] : +- TungstenAggregate(key=[cut#20,color#21], functions=[(avg(cast(price#25 as bigint)),mode=Final,isDistinct=false)], output=[color#21,avg(price)#43]) : +- TungstenExchange hashpartitioning(cut#20,color#21,200), None : +- TungstenAggregate(key=[cut#20,color#21], functions=[(avg(cast(price#25 as bigint)),mode=Partial,isDistinct=false)], output=[cut#20,color#21,sum#58,count#59L]) : +- Scan CsvRelation(-----) +- Sort [color#47 ASC], false, 0 +- TungstenExchange hashpartitioning(color#47,200), None +- ConvertToUnsafe +- Scan CsvRelation(----)
87Spark Streaming from Zinoviev Alexey What’s the difference between Complete and Append output modes?
88Spark Streaming from Zinoviev Alexey COMPLETE, APPEND & UPDATE
89Spark Streaming from Zinoviev Alexey There are two main modes and one in future • append (default)
90Spark Streaming from Zinoviev Alexey There are two main modes and one in future • append (default) • complete
91Spark Streaming from Zinoviev Alexey There are two main modes and one in future • append (default) • complete • update [in dreams]
92Spark Streaming from Zinoviev Alexey Aggregation with watermarks
93Spark Streaming from Zinoviev Alexey SOURCES & SINKS
94Spark Streaming from Zinoviev Alexey Spark Streaming is a brick in the Big Data Wall
95Spark Streaming from Zinoviev Alexey Let's save to Parquet files
96Spark Streaming from Zinoviev Alexey Let's save to Parquet files
97Spark Streaming from Zinoviev Alexey Let's save to Parquet files
98Spark Streaming from Zinoviev Alexey Can we write to Kafka?
99Spark Streaming from Zinoviev Alexey Nightly Build
100Spark Streaming from Zinoviev Alexey Kafka-to-Kafka
101Spark Streaming from Zinoviev Alexey J-K-S-K-S-C
102Spark Streaming from Zinoviev Alexey Pipeline Demo
103Spark Streaming from Zinoviev Alexey I didn’t find sink/source for XXX…
104Spark Streaming from Zinoviev Alexey Console Foreach Sink import org.apache.spark.sql.ForeachWriter val customWriter = new ForeachWriter[String] { override def open(partitionId: Long, version: Long) = true override def process(value: String) = println(value) override def close(errorOrNull: Throwable) = {} } stream.writeStream .queryName(“ForeachOnConsole") .foreach(customWriter) .start
105Spark Streaming from Zinoviev Alexey Pinch of wisdom • check checkpointLocation • don’t use MemoryStream • think about GC pauses • be careful about nighty builds • use .groupBy.count() instead count() • use console sink instead .show() function
106Spark Streaming from Zinoviev Alexey We have no ability… • join two streams • work with update mode • make full outer join • take first N rows • sort without pre-aggregation
107Spark Streaming from Zinoviev Alexey Roadmap 2.2 • Support other data sources (not only S3 + HDFS) • Transactional updates • Dataset is one DSL for all operations • GraphFrames + Structured MLLib • KafkaWriter • TensorFrames
108Spark Streaming from Zinoviev Alexey IN CONCLUSION
109Spark Streaming from Zinoviev Alexey Final point Scalable Fault-Tolerant Real-Time Pipeline with Spark & Kafka is ready for usage
110Spark Streaming from Zinoviev Alexey A few papers about Spark Streaming and Kafka Introduction in Spark + Kafka http://bit.ly/2mJjE4i
111Spark Streaming from Zinoviev Alexey Source code from Demo Spark Tutorial https://github.com/zaleslaw/Spark-Tutorial
112Spark Streaming from Zinoviev Alexey Contacts E-mail : Alexey_Zinovyev@epam.com Twitter : @zaleslaw @BigDataRussia Facebook: https://www.facebook.com/zaleslaw vk.com/big_data_russia Big Data Russia vk.com/java_jvm Java & JVM langs
113Spark Streaming from Zinoviev Alexey Any questions?

More Related Content

PDF
Joker'16 Spark 2 (API changes; Structured Streaming; Encoders)
byAlexey Zinoviev
 
PDF
Beyond the Query – Bringing Complex Access Patterns to NoSQL with DataStax - ...
byStampedeCon
 
PDF
Spark with Elasticsearch - umd version 2014
byHolden Karau
 
PDF
A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise
byPatrick McFadin
 
PDF
Spark with Elasticsearch
byHolden Karau
 
PDF
Integrate Solr with real-time stream processing applications
bylucenerevolution
 
PDF
2014 spark with elastic search
byHenry Saputra
 
PDF
Norikra: Stream Processing with SQL
bySATOSHI TAGOMORI
 
Joker'16 Spark 2 (API changes; Structured Streaming; Encoders)
byAlexey Zinoviev
 
Beyond the Query – Bringing Complex Access Patterns to NoSQL with DataStax - ...
byStampedeCon
 
Spark with Elasticsearch - umd version 2014
byHolden Karau
 
A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise
byPatrick McFadin
 
Spark with Elasticsearch
byHolden Karau
 
Integrate Solr with real-time stream processing applications
bylucenerevolution
 
2014 spark with elastic search
byHenry Saputra
 
Norikra: Stream Processing with SQL
bySATOSHI TAGOMORI
 

What's hot

PDF
Norikra: SQL Stream Processing In Ruby
bySATOSHI TAGOMORI
 
PDF
Building a near real time search engine & analytics for logs using solr
bylucenerevolution
 
PPTX
DataStax: An Introduction to DataStax Enterprise Search
byDataStax Academy
 
PDF
Beyond the Query: A Cassandra + Solr + Spark Love Triangle Using Datastax Ent...
byDataStax Academy
 
PDF
Lightning fast analytics with Spark and Cassandra
bynickmbailey
 
PPTX
Python database interfaces
byMohammad Javad Beheshtian
 
PDF
Cassandra Fundamentals - C* 2.0
byRussell Spitzer
 
PDF
Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...
byLucidworks
 
PDF
Cassandra introduction 2016
byDuyhai Doan
 
PDF
Scalding - the not-so-basics @ ScalaDays 2014
byKonrad Malawski
 
PDF
User Defined Aggregation in Apache Spark: A Love Story
byDatabricks
 
PDF
Spark cassandra connector.API, Best Practices and Use-Cases
byDuyhai Doan
 
PPTX
Solr Search Engine: Optimize Is (Not) Bad for You
bySematext Group, Inc.
 
PDF
Big data 101 for beginners devoxxpl
byDuyhai Doan
 
PDF
HBase RowKey design for Akka Persistence
byKonrad Malawski
 
PDF
Manchester Hadoop Meetup: Cassandra Spark internals
byChristopher Batey
 
PDF
Diagnostics & Debugging webinar
byMongoDB
 
PPTX
Enabling Search in your Cassandra Application with DataStax Enterprise
byDataStax Academy
 
PPTX
Diagnostics and Debugging
byMongoDB
 
PDF
Storing time series data with Apache Cassandra
byPatrick McFadin
 
Norikra: SQL Stream Processing In Ruby
bySATOSHI TAGOMORI
 
Building a near real time search engine & analytics for logs using solr
bylucenerevolution
 
DataStax: An Introduction to DataStax Enterprise Search
byDataStax Academy
 
Beyond the Query: A Cassandra + Solr + Spark Love Triangle Using Datastax Ent...
byDataStax Academy
 
Lightning fast analytics with Spark and Cassandra
bynickmbailey
 
Python database interfaces
byMohammad Javad Beheshtian
 
Cassandra Fundamentals - C* 2.0
byRussell Spitzer
 
Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...
byLucidworks
 
Cassandra introduction 2016
byDuyhai Doan
 
Scalding - the not-so-basics @ ScalaDays 2014
byKonrad Malawski
 
User Defined Aggregation in Apache Spark: A Love Story
byDatabricks
 
Spark cassandra connector.API, Best Practices and Use-Cases
byDuyhai Doan
 
Solr Search Engine: Optimize Is (Not) Bad for You
bySematext Group, Inc.
 
Big data 101 for beginners devoxxpl
byDuyhai Doan
 
HBase RowKey design for Akka Persistence
byKonrad Malawski
 
Manchester Hadoop Meetup: Cassandra Spark internals
byChristopher Batey
 
Diagnostics & Debugging webinar
byMongoDB
 
Enabling Search in your Cassandra Application with DataStax Enterprise
byDataStax Academy
 
Diagnostics and Debugging
byMongoDB
 
Storing time series data with Apache Cassandra
byPatrick McFadin
 

Similar to Kafka pours and Spark resolves

PDF
Spark streaming state of the union
byDatabricks
 
PDF
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
byGuido Schmutz
 
PPTX
Stream, stream, stream: Different streaming methods with Spark and Kafka
byItai Yaffe
 
PPTX
Guest Lecture on Spark Streaming in Stanford CME 323: Distributed Algorithms ...
byTathagata Das
 
PDF
Strata NYC 2015: What's new in Spark Streaming
byDatabricks
 
PDF
Structured Streaming with Kafka
bydatamantra
 
PDF
Building end to end streaming application on Spark
bydatamantra
 
PDF
Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...
byDataStax Academy
 
PPT
Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka
byDataWorks Summit
 
PDF
Databricks Meetup @ Los Angeles Apache Spark User Group
byPaco Nathan
 
PDF
Spark (Structured) Streaming vs. Kafka Streams
byGuido Schmutz
 
PDF
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
byHelena Edelson
 
PDF
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
byGuido Schmutz
 
PDF
Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka
byDatabricks
 
PDF
Introduction to Spark Streaming & Apache Kafka | Big Data Hadoop Spark Tutori...
byCloudxLab
 
PDF
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
byHelena Edelson
 
PDF
NoLambda: Combining Streaming, Ad-Hoc, Machine Learning and Batch Analysis
byHelena Edelson
 
PDF
QCon São Paulo: Real-Time Analytics with Spark Streaming
byPaco Nathan
 
PPTX
Apache Spark Components
byGirish Khanzode
 
PDF
Bellevue Big Data meetup: Dive Deep into Spark Streaming
bySantosh Sahoo
 
Spark streaming state of the union
byDatabricks
 
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
byGuido Schmutz
 
Stream, stream, stream: Different streaming methods with Spark and Kafka
byItai Yaffe
 
Guest Lecture on Spark Streaming in Stanford CME 323: Distributed Algorithms ...
byTathagata Das
 
Strata NYC 2015: What's new in Spark Streaming
byDatabricks
 
Structured Streaming with Kafka
bydatamantra
 
Building end to end streaming application on Spark
bydatamantra
 
Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...
byDataStax Academy
 
Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka
byDataWorks Summit
 
Databricks Meetup @ Los Angeles Apache Spark User Group
byPaco Nathan
 
Spark (Structured) Streaming vs. Kafka Streams
byGuido Schmutz
 
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
byHelena Edelson
 
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
byGuido Schmutz
 
Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka
byDatabricks
 
Introduction to Spark Streaming & Apache Kafka | Big Data Hadoop Spark Tutori...
byCloudxLab
 
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
byHelena Edelson
 
NoLambda: Combining Streaming, Ad-Hoc, Machine Learning and Batch Analysis
byHelena Edelson
 
QCon São Paulo: Real-Time Analytics with Spark Streaming
byPaco Nathan
 
Apache Spark Components
byGirish Khanzode
 
Bellevue Big Data meetup: Dive Deep into Spark Streaming
bySantosh Sahoo
 

More from Alexey Zinoviev

PDF
Java BigData Full Stack Development (version 2.0)
byAlexey Zinoviev
 
PDF
Hadoop Jungle
byAlexey Zinoviev
 
PDF
HappyDev'15 Keynote: Когда все данные станут большими...
byAlexey Zinoviev
 
PDF
Мастер-класс по BigData Tools для HappyDev'15
byAlexey Zinoviev
 
PDF
JavaDayKiev'15 Java in production for Data Mining Research projects
byAlexey Zinoviev
 
PDF
Joker'15 Java straitjackets for MongoDB
byAlexey Zinoviev
 
PDF
JPoint'15 Mom, I so wish Hibernate for my NoSQL database...
byAlexey Zinoviev
 
PDF
Python's slippy path and Tao of thick Pandas: give my data, Rrrrr...
byAlexey Zinoviev
 
PDF
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)
byAlexey Zinoviev
 
PDF
Joker'14 Java as a fundamental working tool of the Data Scientist
byAlexey Zinoviev
 
PDF
First steps in Data Mining Kindergarten
byAlexey Zinoviev
 
PDF
EST: Smart rate (Effective recommendation system for Taxi drivers based on th...
byAlexey Zinoviev
 
PDF
Android Geo Apps in Soviet Russia: Latitude and longitude find you
byAlexey Zinoviev
 
PDF
Keynote on JavaDay Omsk 2014 about new features in Java 8
byAlexey Zinoviev
 
PDF
Big data algorithms and data structures for large scale graphs
byAlexey Zinoviev
 
PDF
"Говнокод-шоу"
byAlexey Zinoviev
 
PDF
Выбор NoSQL базы данных для вашего проекта: "Не в свои сани не садись"
byAlexey Zinoviev
 
PDF
Алгоритмы и структуры данных BigData для графов большой размерности
byAlexey Zinoviev
 
PDF
ALMADA 2013 (computer science school by Yandex and Microsoft Research)
byAlexey Zinoviev
 
PDF
GDG Devfest Omsk 2013. Year of events!
byAlexey Zinoviev
 
Java BigData Full Stack Development (version 2.0)
byAlexey Zinoviev
 
Hadoop Jungle
byAlexey Zinoviev
 
HappyDev'15 Keynote: Когда все данные станут большими...
byAlexey Zinoviev
 
Мастер-класс по BigData Tools для HappyDev'15
byAlexey Zinoviev
 
JavaDayKiev'15 Java in production for Data Mining Research projects
byAlexey Zinoviev
 
Joker'15 Java straitjackets for MongoDB
byAlexey Zinoviev
 
JPoint'15 Mom, I so wish Hibernate for my NoSQL database...
byAlexey Zinoviev
 
Python's slippy path and Tao of thick Pandas: give my data, Rrrrr...
byAlexey Zinoviev
 
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)
byAlexey Zinoviev
 
Joker'14 Java as a fundamental working tool of the Data Scientist
byAlexey Zinoviev
 
First steps in Data Mining Kindergarten
byAlexey Zinoviev
 
EST: Smart rate (Effective recommendation system for Taxi drivers based on th...
byAlexey Zinoviev
 
Android Geo Apps in Soviet Russia: Latitude and longitude find you
byAlexey Zinoviev
 
Keynote on JavaDay Omsk 2014 about new features in Java 8
byAlexey Zinoviev
 
Big data algorithms and data structures for large scale graphs
byAlexey Zinoviev
 
"Говнокод-шоу"
byAlexey Zinoviev
 
Выбор NoSQL базы данных для вашего проекта: "Не в свои сани не садись"
byAlexey Zinoviev
 
Алгоритмы и структуры данных BigData для графов большой размерности
byAlexey Zinoviev
 
ALMADA 2013 (computer science school by Yandex and Microsoft Research)
byAlexey Zinoviev
 
GDG Devfest Omsk 2013. Year of events!
byAlexey Zinoviev
 

Recently uploaded

PDF
Machines, Language & Intelligence: The Evolution of NLP
byayesha butalia
 
PPTX
GrantGreenberg_metagenomic.metagenomics principles and basics
byssuser67fbfc
 
PDF
AWSCertifiedGenerativeAIDeveloper111.pdf
byhemarubini21it
 
PPTX
Decision science introduction in decision making
bymatthewquayson2
 
PPTX
Introduction-to-Microsoft-Excel-by-imran-uos.pptx
byImranHaider416699
 
PDF
BUKTI KEMENANGAN MEMBER KANCAH4D – WD PULUHAN JUTA DARI STARLIGHT PRINCESS!
bykancah4d
 
PPTX
Securing-the-Digital-Frontier saksdjwkdwoj.pptx
bymosami7700
 
PDF
Apache Iceberg - Transactional Storage for Large Datasets
byMichal Gancarski
 
PDF
Nepal's Health Policies: From 1991 to 2024 - A Complete Analysis of Nepal's H...
byPublic Health Concern Nepal
 
PPTX
Agentic AI Presentation | PowerPoint Presentation | Generative AI | Artificia...
bySiddharth Ghosh
 
PPTX
[DSC Europe 25] Slobodan Dolinic - Smart and Intelligent Green Region.pptx
byDataScienceConferenc1
 
PDF
Top 11 Best Sites Buy Gmail Accounts_ A Complete Guide 2025–26.pdf
bynewpvait.com
 
PPTX
[DSC Europe 25] Ivica Milaric - The Future of Gaming and AI Tools.pptx
byDataScienceConferenc1
 
PPTX
Hepatitis.pptx slideshare by me myself and you nknkhk
byPB2013DhanashreeNaik
 
PPTX
[DSC Europe 25] Srba Markovic - From Pilot to Production: Overcoming AI Deplo...
byDataScienceConferenc1
 
PPTX
CC.pptx ffvdv fgfgdrgrgrgbtrgvtygvcrterewe
byPB2013DhanashreeNaik
 
PPTX
_(2) ppt 2021-2022 Protein Metabolism_.pptx
byenghebayousif87
 
PPTX
aaaaaaaaaaaaaaaaaaaaaCloud Computing.pptx
byYashAggarwal233278
 
PPTX
[DSC Europe 25] Mijat Kustudic - Building Financial Intelligence with AI Agen...
byDataScienceConferenc1
 
PPTX
amul_-_business_model.amulbusinessmodelformarketing.pptx
bymanasdidwal03
 
Machines, Language & Intelligence: The Evolution of NLP
byayesha butalia
 
GrantGreenberg_metagenomic.metagenomics principles and basics
byssuser67fbfc
 
AWSCertifiedGenerativeAIDeveloper111.pdf
byhemarubini21it
 
Decision science introduction in decision making
bymatthewquayson2
 
Introduction-to-Microsoft-Excel-by-imran-uos.pptx
byImranHaider416699
 
BUKTI KEMENANGAN MEMBER KANCAH4D – WD PULUHAN JUTA DARI STARLIGHT PRINCESS!
bykancah4d
 
Securing-the-Digital-Frontier saksdjwkdwoj.pptx
bymosami7700
 
Apache Iceberg - Transactional Storage for Large Datasets
byMichal Gancarski
 
Nepal's Health Policies: From 1991 to 2024 - A Complete Analysis of Nepal's H...
byPublic Health Concern Nepal
 
Agentic AI Presentation | PowerPoint Presentation | Generative AI | Artificia...
bySiddharth Ghosh
 
[DSC Europe 25] Slobodan Dolinic - Smart and Intelligent Green Region.pptx
byDataScienceConferenc1
 
Top 11 Best Sites Buy Gmail Accounts_ A Complete Guide 2025–26.pdf
bynewpvait.com
 
[DSC Europe 25] Ivica Milaric - The Future of Gaming and AI Tools.pptx
byDataScienceConferenc1
 
Hepatitis.pptx slideshare by me myself and you nknkhk
byPB2013DhanashreeNaik
 
[DSC Europe 25] Srba Markovic - From Pilot to Production: Overcoming AI Deplo...
byDataScienceConferenc1
 
CC.pptx ffvdv fgfgdrgrgrgbtrgvtygvcrterewe
byPB2013DhanashreeNaik
 
_(2) ppt 2021-2022 Protein Metabolism_.pptx
byenghebayousif87
 
aaaaaaaaaaaaaaaaaaaaaCloud Computing.pptx
byYashAggarwal233278
 
[DSC Europe 25] Mijat Kustudic - Building Financial Intelligence with AI Agen...
byDataScienceConferenc1
 
amul_-_business_model.amulbusinessmodelformarketing.pptx
bymanasdidwal03
 

Kafka pours and Spark resolves

  • 1.
    Kafka pours andSpark resolves! Alexey Zinovyev, Java/BigData Trainer in EPAM
  • 2.
    About With IT since2007 With Java since 2009 With Hadoop since 2012 With Spark since 2014 With EPAM since 2015
  • 3.
    3Spark Streaming fromZinoviev Alexey Contacts E-mail : Alexey_Zinovyev@epam.com Twitter : @zaleslaw @BigDataRussia Facebook: https://www.facebook.com/zaleslaw vk.com/big_data_russia Big Data Russia vk.com/java_jvm Java & JVM langs
  • 4.
    4Spark Streaming fromZinoviev Alexey Spark Family
  • 5.
    5Spark Streaming fromZinoviev Alexey Spark Family
  • 6.
    6Spark Streaming fromZinoviev Alexey Pre-summary • Before RealTime • Spark + Cassandra • Sending messages with Kafka • DStream Kafka Consumer • Structured Streaming in Spark 2.1 • Kafka Writer in Spark 2.2
  • 7.
    7Spark Streaming fromZinoviev Alexey < REAL-TIME
  • 8.
    8Spark Streaming fromZinoviev Alexey Modern Java in 2016Big Data in 2014
  • 9.
    9Spark Streaming fromZinoviev Alexey Batch jobs produce reports. More and more..
  • 10.
    10Spark Streaming fromZinoviev Alexey But customer can wait forever (ok, 1h)
  • 11.
    11Spark Streaming fromZinoviev Alexey Big Data in 2017
  • 12.
    12Spark Streaming fromZinoviev Alexey Machine Learning EVERYWHERE
  • 13.
    13Spark Streaming fromZinoviev Alexey Data Lake in promotional brochure
  • 14.
    14Spark Streaming fromZinoviev Alexey Data Lake in production
  • 15.
    15Spark Streaming fromZinoviev Alexey Simple Flow in Reporting/BI systems
  • 16.
    16Spark Streaming fromZinoviev Alexey Let’s use Spark. It’s fast!
  • 17.
    17Spark Streaming fromZinoviev Alexey MapReduce vs Spark
  • 18.
    18Spark Streaming fromZinoviev Alexey MapReduce vs Spark
  • 19.
    19Spark Streaming fromZinoviev Alexey Simple Flow in Reporting/BI systems with Spark
  • 20.
    20Spark Streaming fromZinoviev Alexey Spark handles last year logs with ease
  • 21.
    21Spark Streaming fromZinoviev Alexey Where can we store events?
  • 22.
    22Spark Streaming fromZinoviev Alexey Let’s use Cassandra to store events!
  • 23.
    23Spark Streaming fromZinoviev Alexey Let’s use Cassandra to read events!
  • 24.
    24Spark Streaming fromZinoviev Alexey Cassandra CREATE KEYSPACE mySpace WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1 }; USE test; CREATE TABLE logs ( application TEXT, time TIMESTAMP, message TEXT, PRIMARY KEY (application, time));
  • 25.
    25Spark Streaming fromZinoviev Alexey Cassandra to Spark val dataSet = sqlContext .read .format("org.apache.spark.sql.cassandra") .options(Map( "table" -> "logs", "keyspace" -> "mySpace" )) .load() dataSet .filter("message = 'Log message'") .show()
  • 26.
    26Spark Streaming fromZinoviev Alexey Simple Flow in Pre-Real-Time systems
  • 27.
    27Spark Streaming fromZinoviev Alexey Spark cluster over Cassandra Cluster
  • 28.
    28Spark Streaming fromZinoviev Alexey More events every second!
  • 29.
    29Spark Streaming fromZinoviev Alexey SENDING MESSAGES
  • 30.
    30Spark Streaming fromZinoviev Alexey Your Father’s Messaging System
  • 31.
    31Spark Streaming fromZinoviev Alexey Your Father’s Messaging System
  • 32.
    32Spark Streaming fromZinoviev Alexey Your Father’s Messaging System InitialContext ctx = new InitialContext(); QueueConnectionFactory f = (QueueConnectionFactory)ctx.lookup(“qCFactory"); QueueConnection con = f.createQueueConnection(); con.start();
  • 33.
    33Spark Streaming fromZinoviev Alexey KAFKA
  • 34.
    34Spark Streaming fromZinoviev Alexey Kafka • messaging system
  • 35.
    35Spark Streaming fromZinoviev Alexey Kafka • messaging system • distributed
  • 36.
    36Spark Streaming fromZinoviev Alexey Kafka • messaging system • distributed • supports Publish-Subscribe model
  • 37.
    37Spark Streaming fromZinoviev Alexey Kafka • messaging system • distributed • supports Publish-Subscribe model • persists messages on disk
  • 38.
    38Spark Streaming fromZinoviev Alexey Kafka • messaging system • distributed • supports Publish-Subscribe model • persists messages on disk • replicates within the cluster (integrated with Zookeeper)
  • 39.
    39Spark Streaming fromZinoviev Alexey The main benefits of Kafka Scalability with zero down time Zero data loss due to replication
  • 40.
    40Spark Streaming fromZinoviev Alexey Kafka Cluster consists of … • brokers (leader or follower) • topics ( >= 1 partition) • partitions • partition offsets • replicas of partition • producers/consumers
  • 41.
    41Spark Streaming fromZinoviev Alexey Kafka Components with topic “messages” #1 Producer Thread #1 Producer Thread #2 Producer Thread #3 Topic: Messages Data Data Data Zookeeper
  • 42.
    42Spark Streaming fromZinoviev Alexey Kafka Components with topic “messages” #2 Producer Thread #1 Producer Thread #2 Producer Thread #3 Topic: Messages Part #1 Part #2 Data Data Data Broker #1 Zookeeper Part #1 Part #2 Leader Leader
  • 43.
    43Spark Streaming fromZinoviev Alexey Kafka Components with topic “messages” #3 Producer Thread #1 Producer Thread #2 Producer Thread #3 Topic: Messages Part #1 Part #2 Data Data Data Broker #1 Broker #2 Zookeeper Part #1 Part #1 Part #2 Part #2
  • 44.
    44Spark Streaming fromZinoviev Alexey Why do we need Zookeeper?
  • 45.
    45Spark Streaming fromZinoviev Alexey Kafka Demo
  • 46.
    46Spark Streaming fromZinoviev Alexey REAL TIME WITH DSTREAMS
  • 47.
    47Spark Streaming fromZinoviev Alexey RDD Factory
  • 48.
    48Spark Streaming fromZinoviev Alexey From socket to console with DStreams
  • 49.
    49Spark Streaming fromZinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
  • 50.
    50Spark Streaming fromZinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
  • 51.
    51Spark Streaming fromZinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
  • 52.
    52Spark Streaming fromZinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
  • 53.
    53Spark Streaming fromZinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
  • 54.
    54Spark Streaming fromZinoviev Alexey Kafka as a main entry point for Spark
  • 55.
    55Spark Streaming fromZinoviev Alexey DStreams Demo
  • 56.
    56Spark Streaming fromZinoviev Alexey How to avoid DStreams with RDD-like API?
  • 57.
    57Spark Streaming fromZinoviev Alexey SPARK 2.2 DISCUSSION
  • 58.
    58Spark Streaming fromZinoviev Alexey Continuous Applications
  • 59.
    59Spark Streaming fromZinoviev Alexey Continuous Applications cases • Updating data that will be served in real time • Extract, transform and load (ETL) • Creating a real-time version of an existing batch job • Online machine learning
  • 60.
    60Spark Streaming fromZinoviev Alexey The main concept of Structured Streaming You can express your streaming computation the same way you would express a batch computation on static data.
  • 61.
    61Spark Streaming fromZinoviev Alexey Batch Spark 2.2 // Read JSON once from S3 logsDF = spark.read.json("s3://logs") // Transform with DataFrame API and save logsDF.select("user", "url", "date") .write.parquet("s3://out")
  • 62.
    62Spark Streaming fromZinoviev Alexey Real Time Spark 2.2 // Read JSON continuously from S3 logsDF = spark.readStream.json("s3://logs") // Transform with DataFrame API and save logsDF.select("user", "url", "date") .writeStream.parquet("s3://out") .start()
  • 63.
    63Spark Streaming fromZinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
  • 64.
    64Spark Streaming fromZinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
  • 65.
    65Spark Streaming fromZinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
  • 66.
    66Spark Streaming fromZinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count() Don’t forget to start Streaming
  • 67.
    67Spark Streaming fromZinoviev Alexey Unlimited Table
  • 68.
    68Spark Streaming fromZinoviev Alexey WordCount with Structured Streaming [Complete Mode]
  • 69.
    69Spark Streaming fromZinoviev Alexey Kafka -> Structured Streaming -> Console
  • 70.
    70Spark Streaming fromZinoviev Alexey Kafka To Console Demo
  • 71.
    71Spark Streaming fromZinoviev Alexey OPERATIONS
  • 72.
    72Spark Streaming fromZinoviev Alexey You can … • filter • sort • aggregate • join • foreach • explain
  • 73.
    73Spark Streaming fromZinoviev Alexey Operators Demo
  • 74.
    74Spark Streaming fromZinoviev Alexey How it works?
  • 75.
    75Spark Streaming fromZinoviev Alexey Deep Diving in Spark Internals Dataset
  • 76.
    76Spark Streaming fromZinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan
  • 77.
    77Spark Streaming fromZinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan Optimized Logical Plan
  • 78.
    78Spark Streaming fromZinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan Optimized Logical Plan Logical Plan Logical Plan Logical PlanPhysical Plan
  • 79.
    79Spark Streaming fromZinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan Optimized Logical Plan Logical Plan Logical Plan Logical PlanPhysical Plan Selected Physical Plan Planner
  • 80.
    80Spark Streaming fromZinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan Optimized Logical Plan Logical Plan Logical Plan Logical PlanPhysical Plan Selected Physical Plan
  • 81.
    81Spark Streaming fromZinoviev Alexey Deep Diving in Spark Internals Dataset Logical Plan Optimized Logical Plan Logical Plan Logical Plan Logical PlanPhysical Plan Selected Physical Plan Incremental #1 Incremental #2 Incremental #3
  • 82.
    82Spark Streaming fromZinoviev Alexey Incremental Execution: Planner polls Planner
  • 83.
    83Spark Streaming fromZinoviev Alexey Incremental Execution: Planner runs Planner Offset: 1 - 5 Incremental #1 R U N Count: 5
  • 84.
    84Spark Streaming fromZinoviev Alexey Incremental Execution: Planner runs #2 Planner Offset: 1 - 5 Incremental #1 Incremental #2 Offset: 6 - 9 R U N Count: 5 Count: 4
  • 85.
    85Spark Streaming fromZinoviev Alexey Aggregation with State Planner Offset: 1 - 5 Incremental #1 Incremental #2 Offset: 6 - 9 R U N Count: 5 Count: 5+4=9 5
  • 86.
    86Spark Streaming fromZinoviev Alexey DataSet.explain() == Physical Plan == Project [avg(price)#43,carat#45] +- SortMergeJoin [color#21], [color#47] :- Sort [color#21 ASC], false, 0 : +- TungstenExchange hashpartitioning(color#21,200), None : +- Project [avg(price)#43,color#21] : +- TungstenAggregate(key=[cut#20,color#21], functions=[(avg(cast(price#25 as bigint)),mode=Final,isDistinct=false)], output=[color#21,avg(price)#43]) : +- TungstenExchange hashpartitioning(cut#20,color#21,200), None : +- TungstenAggregate(key=[cut#20,color#21], functions=[(avg(cast(price#25 as bigint)),mode=Partial,isDistinct=false)], output=[cut#20,color#21,sum#58,count#59L]) : +- Scan CsvRelation(-----) +- Sort [color#47 ASC], false, 0 +- TungstenExchange hashpartitioning(color#47,200), None +- ConvertToUnsafe +- Scan CsvRelation(----)
  • 87.
    87Spark Streaming fromZinoviev Alexey What’s the difference between Complete and Append output modes?
  • 88.
    88Spark Streaming fromZinoviev Alexey COMPLETE, APPEND & UPDATE
  • 89.
    89Spark Streaming fromZinoviev Alexey There are two main modes and one in future • append (default)
  • 90.
    90Spark Streaming fromZinoviev Alexey There are two main modes and one in future • append (default) • complete
  • 91.
    91Spark Streaming fromZinoviev Alexey There are two main modes and one in future • append (default) • complete • update [in dreams]
  • 92.
    92Spark Streaming fromZinoviev Alexey Aggregation with watermarks
  • 93.
    93Spark Streaming fromZinoviev Alexey SOURCES & SINKS
  • 94.
    94Spark Streaming fromZinoviev Alexey Spark Streaming is a brick in the Big Data Wall
  • 95.
    95Spark Streaming fromZinoviev Alexey Let's save to Parquet files
  • 96.
    96Spark Streaming fromZinoviev Alexey Let's save to Parquet files
  • 97.
    97Spark Streaming fromZinoviev Alexey Let's save to Parquet files
  • 98.
    98Spark Streaming fromZinoviev Alexey Can we write to Kafka?
  • 99.
    99Spark Streaming fromZinoviev Alexey Nightly Build
  • 100.
    100Spark Streaming fromZinoviev Alexey Kafka-to-Kafka
  • 101.
    101Spark Streaming fromZinoviev Alexey J-K-S-K-S-C
  • 102.
    102Spark Streaming fromZinoviev Alexey Pipeline Demo
  • 103.
    103Spark Streaming fromZinoviev Alexey I didn’t find sink/source for XXX…
  • 104.
    104Spark Streaming fromZinoviev Alexey Console Foreach Sink import org.apache.spark.sql.ForeachWriter val customWriter = new ForeachWriter[String] { override def open(partitionId: Long, version: Long) = true override def process(value: String) = println(value) override def close(errorOrNull: Throwable) = {} } stream.writeStream .queryName(“ForeachOnConsole") .foreach(customWriter) .start
  • 105.
    105Spark Streaming fromZinoviev Alexey Pinch of wisdom • check checkpointLocation • don’t use MemoryStream • think about GC pauses • be careful about nighty builds • use .groupBy.count() instead count() • use console sink instead .show() function
  • 106.
    106Spark Streaming fromZinoviev Alexey We have no ability… • join two streams • work with update mode • make full outer join • take first N rows • sort without pre-aggregation
  • 107.
    107Spark Streaming fromZinoviev Alexey Roadmap 2.2 • Support other data sources (not only S3 + HDFS) • Transactional updates • Dataset is one DSL for all operations • GraphFrames + Structured MLLib • KafkaWriter • TensorFrames
  • 108.
    108Spark Streaming fromZinoviev Alexey IN CONCLUSION
  • 109.
    109Spark Streaming fromZinoviev Alexey Final point Scalable Fault-Tolerant Real-Time Pipeline with Spark & Kafka is ready for usage
  • 110.
    110Spark Streaming fromZinoviev Alexey A few papers about Spark Streaming and Kafka Introduction in Spark + Kafka http://bit.ly/2mJjE4i
  • 111.
    111Spark Streaming fromZinoviev Alexey Source code from Demo Spark Tutorial https://github.com/zaleslaw/Spark-Tutorial
  • 112.
    112Spark Streaming fromZinoviev Alexey Contacts E-mail : Alexey_Zinovyev@epam.com Twitter : @zaleslaw @BigDataRussia Facebook: https://www.facebook.com/zaleslaw vk.com/big_data_russia Big Data Russia vk.com/java_jvm Java & JVM langs
  • 113.
    113Spark Streaming fromZinoviev Alexey Any questions?