Alexey Zinoviev, profile picture
Uploaded byAlexey Zinoviev
PDF, PPTX844 views

Joker'16 Spark 2 (API changes; Structured Streaming; Encoders)

This document presents an overview of Spark 2 and its functionalities, including programmatic examples for loading data, using RDDs, and structured streaming. It discusses concepts such as map-reduce, machine learning, and differences between Spark and traditional Hadoop approaches. Additionally, it highlights the evolution of Spark APIs and the significance of Spark sessions in managing data frameworks.

Embed presentation

Download as PDF, PPTX
1 / 142
2 / 142
3 / 142
4 / 142
5 / 142
6 / 142
7 / 142
8 / 142
9 / 142
10 / 142
11 / 142
12 / 142
13 / 142
14 / 142
15 / 142
16 / 142
17 / 142
18 / 142
19 / 142
20 / 142
21 / 142
22 / 142
23 / 142
24 / 142
25 / 142
26 / 142
27 / 142
28 / 142
29 / 142
30 / 142
31 / 142
32 / 142
33 / 142
34 / 142
35 / 142
36 / 142
37 / 142
38 / 142
39 / 142
40 / 142
41 / 142
42 / 142
43 / 142
44 / 142
45 / 142
46 / 142
47 / 142
48 / 142
49 / 142
50 / 142
51 / 142
52 / 142
53 / 142
54 / 142
55 / 142
56 / 142
57 / 142
58 / 142
59 / 142
60 / 142
61 / 142
62 / 142
63 / 142
64 / 142
65 / 142
66 / 142
67 / 142
68 / 142
69 / 142
70 / 142
71 / 142
72 / 142
73 / 142
74 / 142
75 / 142
76 / 142
77 / 142
78 / 142
79 / 142
80 / 142
81 / 142
82 / 142
83 / 142
84 / 142
85 / 142
86 / 142
87 / 142
88 / 142
89 / 142
90 / 142
91 / 142
92 / 142
93 / 142
94 / 142
95 / 142
96 / 142
97 / 142
98 / 142
99 / 142
100 / 142
101 / 142
102 / 142
103 / 142
104 / 142
105 / 142
106 / 142
107 / 142
108 / 142
109 / 142
110 / 142
111 / 142
112 / 142
113 / 142
114 / 142
115 / 142
116 / 142
117 / 142
118 / 142
119 / 142
120 / 142
121 / 142
122 / 142
123 / 142
124 / 142
125 / 142
126 / 142
127 / 142
128 / 142
129 / 142
130 / 142
131 / 142
132 / 142
133 / 142
134 / 142
135 / 142
136 / 142
137 / 142
138 / 142
139 / 142
140 / 142
141 / 142
142 / 142
Spark 2 Alexey Zinovyev, Java/BigData Trainer in EPAM
About With IT since 2007 With Java since 2009 With Hadoop since 2012 With EPAM since 2015
3Spark 2 from Zinoviev Alexey Contacts E-mail : Alexey_Zinovyev@epam.com Twitter : @zaleslaw @BigDataRussia Facebook: https://www.facebook.com/zaleslaw vk.com/big_data_russia Big Data Russia vk.com/java_jvm Java & JVM langs
4Spark 2 from Zinoviev Alexey Sprk Dvlprs! Let’s start!
5Spark 2 from Zinoviev Alexey < SPARK 2.0
6Spark 2 from Zinoviev Alexey Modern Java in 2016Big Data in 2014
7Spark 2 from Zinoviev Alexey Big Data in 2017
8Spark 2 from Zinoviev Alexey Machine Learning EVERYWHERE
9Spark 2 from Zinoviev Alexey Machine Learning vs Traditional Programming
10Spark 2 from Zinoviev Alexey
11Spark 2 from Zinoviev Alexey Something wrong with HADOOP
12Spark 2 from Zinoviev Alexey Hadoop is not SEXY
13Spark 2 from Zinoviev Alexey Whaaaat?
14Spark 2 from Zinoviev Alexey Map Reduce Job Writing
15Spark 2 from Zinoviev Alexey MR code
16Spark 2 from Zinoviev Alexey Hadoop Developers Right Now
17Spark 2 from Zinoviev Alexey Iterative Calculations 10x – 100x
18Spark 2 from Zinoviev Alexey MapReduce vs Spark
19Spark 2 from Zinoviev Alexey MapReduce vs Spark
20Spark 2 from Zinoviev Alexey MapReduce vs Spark
21Spark 2 from Zinoviev Alexey SPARK INTRO
22Spark 2 from Zinoviev Alexey Loading val localData = (5,7,1,12,10,25) val ourFirstRDD = sc.parallelize(localData) val textFile = sc.textFile("hdfs://...")
23Spark 2 from Zinoviev Alexey Loading val localData = (5,7,1,12,10,25) val ourFirstRDD = sc.parallelize(localData) val textFile = sc.textFile("hdfs://...")
24Spark 2 from Zinoviev Alexey Loading val localData = (5,7,1,12,10,25) val ourFirstRDD = sc.parallelize(localData) // from file val textFile = sc.textFile("hdfs://...")
25Spark 2 from Zinoviev Alexey Loading val localData = (5,7,1,12,10,25) val ourFirstRDD = sc.parallelize(localData) // from file val textFile = sc.textFile("hdfs://...")
26Spark 2 from Zinoviev Alexey Loading val localData = (5,7,1,12,10,25) val ourFirstRDD = sc.parallelize(localData) // from file val textFile = sc.textFile("hdfs://...")
27Spark 2 from Zinoviev Alexey Word Count val textFile = sc.textFile("hdfs://...") val counts = textFile .flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs://...")
28Spark 2 from Zinoviev Alexey Word Count val textFile = sc.textFile("hdfs://...") val counts = textFile .flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs://...")
29Spark 2 from Zinoviev Alexey Word Count val textFile = sc.textFile("hdfs://...") val counts = textFile .flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs://...")
30Spark 2 from Zinoviev Alexey SQL val hive = new HiveContext(spark) hive.hql(“CREATE TABLE IF NOT EXISTS src (key INT, value STRING)”) hive.hql(“LOAD DATA LOCAL INPATH ‘…/kv1.txt’ INTO TABLE src”) val results = hive.hql(“FROM src SELECT key, value”).collect()
31Spark 2 from Zinoviev Alexey SQL val hive = new HiveContext(spark) hive.hql(“CREATE TABLE IF NOT EXISTS src (key INT, value STRING)”) hive.hql(“LOAD DATA LOCAL INPATH ‘…/kv1.txt’ INTO TABLE src”) val results = hive.hql(“FROM src SELECT key, value”).collect()
32Spark 2 from Zinoviev Alexey SQL val hive = new HiveContext(spark) hive.hql(“CREATE TABLE IF NOT EXISTS src (key INT, value STRING)”) hive.hql(“LOAD DATA LOCAL INPATH ‘…/kv1.txt’ INTO TABLE src”) val results = hive.hql(“FROM src SELECT key, value”).collect()
33Spark 2 from Zinoviev Alexey SQL val hive = new HiveContext(spark) hive.hql(“CREATE TABLE IF NOT EXISTS src (key INT, value STRING)”) hive.hql(“LOAD DATA LOCAL INPATH ‘…/kv1.txt’ INTO TABLE src”) val results = hive.hql(“FROM src SELECT key, value”).collect()
34Spark 2 from Zinoviev Alexey RDD Factory
35Spark 2 from Zinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
36Spark 2 from Zinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
37Spark 2 from Zinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
38Spark 2 from Zinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
39Spark 2 from Zinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
40Spark 2 from Zinoviev Alexey It’s very easy to understand Graph algorithms def pregel[A](initialMsg: A, maxIterations: Int, activeDirection: EdgeDirection)( vprog: (VertexID, VD, A) => VD, sendMsg: EdgeTriplet[VD, ED] => Iterator[(VertexID,A)], mergeMsg: (A, A) => A) : Graph[VD, ED] = { <Your Pregel Computations> }
41Spark 2 from Zinoviev Alexey SPARK 2.0 DISCUSSION
42Spark 2 from Zinoviev Alexey Spark Family
43Spark 2 from Zinoviev Alexey Spark Family
44Spark 2 from Zinoviev Alexey Case #0 : How to avoid DStreams with RDD-like API?
45Spark 2 from Zinoviev Alexey Continuous Applications
46Spark 2 from Zinoviev Alexey Continuous Applications cases • Updating data that will be served in real time • Extract, transform and load (ETL) • Creating a real-time version of an existing batch job • Online machine learning
47Spark 2 from Zinoviev Alexey Write Batches
48Spark 2 from Zinoviev Alexey Catch Streaming
49Spark 2 from Zinoviev Alexey The main concept of Structured Streaming You can express your streaming computation the same way you would express a batch computation on static data.
50Spark 2 from Zinoviev Alexey Batch // Read JSON once from S3 logsDF = spark.read.json("s3://logs") // Transform with DataFrame API and save logsDF.select("user", "url", "date") .write.parquet("s3://out")
51Spark 2 from Zinoviev Alexey Real Time // Read JSON continuously from S3 logsDF = spark.readStream.json("s3://logs") // Transform with DataFrame API and save logsDF.select("user", "url", "date") .writeStream.parquet("s3://out") .start()
52Spark 2 from Zinoviev Alexey Unlimited Table
53Spark 2 from Zinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
54Spark 2 from Zinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
55Spark 2 from Zinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
56Spark 2 from Zinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count() Don’t forget to start Streaming
57Spark 2 from Zinoviev Alexey WordCount with Structured Streaming
58Spark 2 from Zinoviev Alexey Structured Streaming provides … • fast & scalable • fault-tolerant • end-to-end with exactly-once semantic • stream processing • ability to use DataFrame/DataSet API for streaming
59Spark 2 from Zinoviev Alexey Structured Streaming provides (in dreams) … • fast & scalable • fault-tolerant • end-to-end with exactly-once semantic • stream processing • ability to use DataFrame/DataSet API for streaming
60Spark 2 from Zinoviev Alexey Let’s UNION streaming and static DataSets
61Spark 2 from Zinoviev Alexey Let’s UNION streaming and static DataSets org.apache.spark.sql. AnalysisException: Union between streaming and batch DataFrames/Datasets is not supported;
62Spark 2 from Zinoviev Alexey Let’s UNION streaming and static DataSets Go to UnsupportedOperationChecker.scala and check your operation
63Spark 2 from Zinoviev Alexey Case #1 : We should think about optimization in RDD terms
64Spark 2 from Zinoviev Alexey Single Thread collection
65Spark 2 from Zinoviev Alexey No perf issues, right?
66Spark 2 from Zinoviev Alexey The main concept more partitions = more parallelism
67Spark 2 from Zinoviev Alexey Do it parallel
68Spark 2 from Zinoviev Alexey I’d like NARROW
69Spark 2 from Zinoviev Alexey Map, filter, filter
70Spark 2 from Zinoviev Alexey GroupByKey, join
71Spark 2 from Zinoviev Alexey Case #2 : DataFrames suggest mix SQL and Scala functions
72Spark 2 from Zinoviev Alexey History of Spark APIs
73Spark 2 from Zinoviev Alexey RDD rdd.filter(_.age > 21) // RDD df.filter("age > 21") // DataFrame SQL-style df.filter(df.col("age").gt(21)) // Expression style dataset.filter(_.age < 21); // Dataset API
74Spark 2 from Zinoviev Alexey History of Spark APIs
75Spark 2 from Zinoviev Alexey SQL rdd.filter(_.age > 21) // RDD df.filter("age > 21") // DataFrame SQL-style df.filter(df.col("age").gt(21)) // Expression style dataset.filter(_.age < 21); // Dataset API
76Spark 2 from Zinoviev Alexey Expression rdd.filter(_.age > 21) // RDD df.filter("age > 21") // DataFrame SQL-style df.filter(df.col("age").gt(21)) // Expression style dataset.filter(_.age < 21); // Dataset API
77Spark 2 from Zinoviev Alexey History of Spark APIs
78Spark 2 from Zinoviev Alexey DataSet rdd.filter(_.age > 21) // RDD df.filter("age > 21") // DataFrame SQL-style df.filter(df.col("age").gt(21)) // Expression style dataset.filter(_.age < 21); // Dataset API
79Spark 2 from Zinoviev Alexey Case #2 : DataFrame is referring to data attributes by name
80Spark 2 from Zinoviev Alexey DataSet = RDD’s types + DataFrame’s Catalyst • RDD API • compile-time type-safety • off-heap storage mechanism • performance benefits of the Catalyst query optimizer • Tungsten
81Spark 2 from Zinoviev Alexey DataSet = RDD’s types + DataFrame’s Catalyst • RDD API • compile-time type-safety • off-heap storage mechanism • performance benefits of the Catalyst query optimizer • Tungsten
82Spark 2 from Zinoviev Alexey Structured APIs in SPARK
83Spark 2 from Zinoviev Alexey Unified API in Spark 2.0 DataFrame = Dataset[Row] Dataframe is a schemaless (untyped) Dataset now
84Spark 2 from Zinoviev Alexey Define case class case class User(email: String, footSize: Long, name: String) // DataFrame -> DataSet with Users val userDS = spark.read.json("/home/tmp/datasets/users.json").as[User] userDS.map(_.name).collect() userDS.filter(_.footSize > 38).collect() ds.rdd // IF YOU REALLY WANT
85Spark 2 from Zinoviev Alexey Read JSON case class User(email: String, footSize: Long, name: String) // DataFrame -> DataSet with Users val userDS = spark.read.json("/home/tmp/datasets/users.json").as[User] userDS.map(_.name).collect() userDS.filter(_.footSize > 38).collect() ds.rdd // IF YOU REALLY WANT
86Spark 2 from Zinoviev Alexey Filter by Field case class User(email: String, footSize: Long, name: String) // DataFrame -> DataSet with Users val userDS = spark.read.json("/home/tmp/datasets/users.json").as[User] userDS.map(_.name).collect() userDS.filter(_.footSize > 38).collect() ds.rdd // IF YOU REALLY WANT
87Spark 2 from Zinoviev Alexey Case #3 : Spark has many contexts
88Spark 2 from Zinoviev Alexey Spark Session • New entry point in spark for creating datasets • Replaces SQLContext, HiveContext and StreamingContext • Move from SparkContext to SparkSession signifies move away from RDD
89Spark 2 from Zinoviev Alexey Spark Session val sparkSession = SparkSession.builder .master("local") .appName("spark session example") .getOrCreate() val df = sparkSession.read .option("header","true") .csv("src/main/resources/names.csv") df.show()
90Spark 2 from Zinoviev Alexey No, I want to create my lovely RDDs
91Spark 2 from Zinoviev Alexey Where’s parallelize() method?
92Spark 2 from Zinoviev Alexey RDD? case class User(email: String, footSize: Long, name: String) // DataFrame -> DataSet with Users val userDS = spark.read.json("/home/tmp/datasets/users.json").as[User] userDS.map(_.name).collect() userDS.filter(_.footSize > 38).collect() ds.rdd // IF YOU REALLY WANT
93Spark 2 from Zinoviev Alexey Case #4 : Spark uses Java serialization A LOT
94Spark 2 from Zinoviev Alexey Two choices to distribute data across cluster • Java serialization By default with ObjectOutputStream • Kryo serialization Should register classes (no support of Serialazible)
95Spark 2 from Zinoviev Alexey The main problem: overhead of serializing Each serialized object contains the class structure as well as the values
96Spark 2 from Zinoviev Alexey The main problem: overhead of serializing Each serialized object contains the class structure as well as the values Don’t forget about GC
97Spark 2 from Zinoviev Alexey Tungsten Compact Encoding
98Spark 2 from Zinoviev Alexey Encoder’s concept Generate bytecode to interact with off-heap & Give access to attributes without ser/deser
99Spark 2 from Zinoviev Alexey Encoders
100Spark 2 from Zinoviev Alexey No custom encoders
101Spark 2 from Zinoviev Alexey Case #5 : Not enough storage levels 
102Spark 2 from Zinoviev Alexey Caching in Spark • Frequently used RDD can be stored in memory • One method, one short-cut: persist(), cache() • SparkContext keeps track of cached RDD • Serialized or deserialized Java objects
103Spark 2 from Zinoviev Alexey Full list of options • MEMORY_ONLY • MEMORY_AND_DISK • MEMORY_ONLY_SER • MEMORY_AND_DISK_SER • DISK_ONLY • MEMORY_ONLY_2, MEMORY_AND_DISK_2
104Spark 2 from Zinoviev Alexey Spark Core Storage Level • MEMORY_ONLY (default for Spark Core) • MEMORY_AND_DISK • MEMORY_ONLY_SER • MEMORY_AND_DISK_SER • DISK_ONLY • MEMORY_ONLY_2, MEMORY_AND_DISK_2
105Spark 2 from Zinoviev Alexey Spark Streaming Storage Level • MEMORY_ONLY (default for Spark Core) • MEMORY_AND_DISK • MEMORY_ONLY_SER (default for Spark Streaming) • MEMORY_AND_DISK_SER • DISK_ONLY • MEMORY_ONLY_2, MEMORY_AND_DISK_2
106Spark 2 from Zinoviev Alexey Developer API to make new Storage Levels
107Spark 2 from Zinoviev Alexey What’s the most popular file format in BigData?
108Spark 2 from Zinoviev Alexey Case #6 : External libraries to read CSV
109Spark 2 from Zinoviev Alexey Easy to read CSV data = sqlContext.read.format("csv") .option("header", "true") .option("inferSchema", "true") .load("/datasets/samples/users.csv") data.cache() data.createOrReplaceTempView(“users") display(data)
110Spark 2 from Zinoviev Alexey Case #7 : How to measure Spark performance?
111Spark 2 from Zinoviev Alexey You'd measure performance!
112Spark 2 from Zinoviev Alexey TPCDS 99 Queries http://bit.ly/2dObMsH
113Spark 2 from Zinoviev Alexey
114Spark 2 from Zinoviev Alexey How to benchmark Spark
115Spark 2 from Zinoviev Alexey Special Tool from Databricks Benchmark Tool for SparkSQL https://github.com/databricks/spark-sql-perf
116Spark 2 from Zinoviev Alexey Spark 2 vs Spark 1.6
117Spark 2 from Zinoviev Alexey Case #8 : What’s faster: SQL or DataSet API?
118Spark 2 from Zinoviev Alexey Job Stages in old Spark
119Spark 2 from Zinoviev Alexey Scheduler Optimizations
120Spark 2 from Zinoviev Alexey Catalyst Optimizer for DataFrames
121Spark 2 from Zinoviev Alexey Unified Logical Plan DataFrame = Dataset[Row]
122Spark 2 from Zinoviev Alexey Bytecode
123Spark 2 from Zinoviev Alexey DataSet.explain() == Physical Plan == Project [avg(price)#43,carat#45] +- SortMergeJoin [color#21], [color#47] :- Sort [color#21 ASC], false, 0 : +- TungstenExchange hashpartitioning(color#21,200), None : +- Project [avg(price)#43,color#21] : +- TungstenAggregate(key=[cut#20,color#21], functions=[(avg(cast(price#25 as bigint)),mode=Final,isDistinct=false)], output=[color#21,avg(price)#43]) : +- TungstenExchange hashpartitioning(cut#20,color#21,200), None : +- TungstenAggregate(key=[cut#20,color#21], functions=[(avg(cast(price#25 as bigint)),mode=Partial,isDistinct=false)], output=[cut#20,color#21,sum#58,count#59L]) : +- Scan CsvRelation(-----) +- Sort [color#47 ASC], false, 0 +- TungstenExchange hashpartitioning(color#47,200), None +- ConvertToUnsafe +- Scan CsvRelation(----)
124Spark 2 from Zinoviev Alexey Case #9 : Why does explain() show so many Tungsten things?
125Spark 2 from Zinoviev Alexey How to be effective with CPU • Runtime code generation • Exploiting cache locality • Off-heap memory management
126Spark 2 from Zinoviev Alexey Tungsten’s goal Push performance closer to the limits of modern hardware
127Spark 2 from Zinoviev Alexey Maybe something UNSAFE?
128Spark 2 from Zinoviev Alexey UnsafeRowFormat • Bit set for tracking null values • Small values are inlined • For variable-length values are stored relative offset into the variablelength data section • Rows are always 8-byte word aligned • Equality comparison and hashing can be performed on raw bytes without requiring additional interpretation
129Spark 2 from Zinoviev Alexey Case #10 : Can I influence on Memory Management in Spark?
130Spark 2 from Zinoviev Alexey Case #11 : Should I tune generation’s stuff?
131Spark 2 from Zinoviev Alexey Cached Data
132Spark 2 from Zinoviev Alexey During operations
133Spark 2 from Zinoviev Alexey For your needs
134Spark 2 from Zinoviev Alexey For Dark Lord
135Spark 2 from Zinoviev Alexey IN CONCLUSION
136Spark 2 from Zinoviev Alexey We have no ability… • join structured streaming and other sources to handle it • one unified ML API • GraphX rethinking and redesign • Custom encoders • Datasets everywhere • integrate with something important
137Spark 2 from Zinoviev Alexey Roadmap • Support other data sources (not only S3 + HDFS) • Transactional updates • Dataset is one DSL for all operations • GraphFrames + Structured MLLib • Tungsten: custom encoders • The RDD-based API is expected to be removed in Spark 3.0
138Spark 2 from Zinoviev Alexey And we can DO IT! Real-Time Data-Marts Batch Data-Marts Relations Graph Ontology Metadata Search Index Events & Alarms Real-time Dashboarding Events & Alarms All Raw Data backup is stored here Real-time Data Ingestion Batch Data Ingestion Real-Time ETL & CEP Batch ETL & Raw Area Scheduler Internal External Social HDFS → CFS as an option Time-Series Data Titan & KairosDB store data in Cassandra Push Events & Alarms (Email, SNMP etc.)
139Spark 2 from Zinoviev Alexey First Part
140Spark 2 from Zinoviev Alexey Second Part
141Spark 2 from Zinoviev Alexey Contacts E-mail : Alexey_Zinovyev@epam.com Twitter : @zaleslaw @BigDataRussia Facebook: https://www.facebook.com/zaleslaw vk.com/big_data_russia Big Data Russia vk.com/java_jvm Java & JVM langs
142Spark 2 from Zinoviev Alexey Any questions?

More Related Content

PDF
Kafka pours and Spark resolves
byAlexey Zinoviev
 
PDF
Spark with Elasticsearch - umd version 2014
byHolden Karau
 
PDF
Integrate Solr with real-time stream processing applications
bylucenerevolution
 
PDF
Spark with Elasticsearch
byHolden Karau
 
PDF
Beyond the Query – Bringing Complex Access Patterns to NoSQL with DataStax - ...
byStampedeCon
 
PDF
A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise
byPatrick McFadin
 
PDF
2014 spark with elastic search
byHenry Saputra
 
PDF
Building a near real time search engine & analytics for logs using solr
bylucenerevolution
 
Kafka pours and Spark resolves
byAlexey Zinoviev
 
Spark with Elasticsearch - umd version 2014
byHolden Karau
 
Integrate Solr with real-time stream processing applications
bylucenerevolution
 
Spark with Elasticsearch
byHolden Karau
 
Beyond the Query – Bringing Complex Access Patterns to NoSQL with DataStax - ...
byStampedeCon
 
A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise
byPatrick McFadin
 
2014 spark with elastic search
byHenry Saputra
 
Building a near real time search engine & analytics for logs using solr
bylucenerevolution
 

What's hot

PDF
Beyond the Query: A Cassandra + Solr + Spark Love Triangle Using Datastax Ent...
byDataStax Academy
 
PDF
Diagnostics & Debugging webinar
byMongoDB
 
PDF
User Defined Aggregation in Apache Spark: A Love Story
byDatabricks
 
PPTX
Diagnostics and Debugging
byMongoDB
 
PDF
Norikra: SQL Stream Processing In Ruby
bySATOSHI TAGOMORI
 
PPTX
DataStax: An Introduction to DataStax Enterprise Search
byDataStax Academy
 
PPTX
Enabling Search in your Cassandra Application with DataStax Enterprise
byDataStax Academy
 
PPTX
Solr Search Engine: Optimize Is (Not) Bad for You
bySematext Group, Inc.
 
PPTX
Data Wars: The Bloody Enterprise strikes back
byVictor_Cr
 
PDF
Spark cassandra connector.API, Best Practices and Use-Cases
byDuyhai Doan
 
PPTX
Python database interfaces
byMohammad Javad Beheshtian
 
PDF
Scalding - the not-so-basics @ ScalaDays 2014
byKonrad Malawski
 
PDF
Lightning fast analytics with Spark and Cassandra
bynickmbailey
 
PPTX
Beyond parallelize and collect - Spark Summit East 2016
byHolden Karau
 
PDF
HBase RowKey design for Akka Persistence
byKonrad Malawski
 
PDF
Norikra: Stream Processing with SQL
bySATOSHI TAGOMORI
 
PDF
Apache cassandra and spark. you got the the lighter, let's start the fire
byPatrick McFadin
 
PDF
Cassandra Fundamentals - C* 2.0
byRussell Spitzer
 
PDF
Cassandra introduction 2016
byDuyhai Doan
 
PDF
Big data 101 for beginners devoxxpl
byDuyhai Doan
 
Beyond the Query: A Cassandra + Solr + Spark Love Triangle Using Datastax Ent...
byDataStax Academy
 
Diagnostics & Debugging webinar
byMongoDB
 
User Defined Aggregation in Apache Spark: A Love Story
byDatabricks
 
Diagnostics and Debugging
byMongoDB
 
Norikra: SQL Stream Processing In Ruby
bySATOSHI TAGOMORI
 
DataStax: An Introduction to DataStax Enterprise Search
byDataStax Academy
 
Enabling Search in your Cassandra Application with DataStax Enterprise
byDataStax Academy
 
Solr Search Engine: Optimize Is (Not) Bad for You
bySematext Group, Inc.
 
Data Wars: The Bloody Enterprise strikes back
byVictor_Cr
 
Spark cassandra connector.API, Best Practices and Use-Cases
byDuyhai Doan
 
Python database interfaces
byMohammad Javad Beheshtian
 
Scalding - the not-so-basics @ ScalaDays 2014
byKonrad Malawski
 
Lightning fast analytics with Spark and Cassandra
bynickmbailey
 
Beyond parallelize and collect - Spark Summit East 2016
byHolden Karau
 
HBase RowKey design for Akka Persistence
byKonrad Malawski
 
Norikra: Stream Processing with SQL
bySATOSHI TAGOMORI
 
Apache cassandra and spark. you got the the lighter, let's start the fire
byPatrick McFadin
 
Cassandra Fundamentals - C* 2.0
byRussell Spitzer
 
Cassandra introduction 2016
byDuyhai Doan
 
Big data 101 for beginners devoxxpl
byDuyhai Doan
 

Viewers also liked

PDF
Spark Summit EU talk by Ted Malaska
bySpark Summit
 
PPTX
Keeping Spark on Track: Productionizing Spark for ETL
byDatabricks
 
PDF
Data Source API in Spark
byDatabricks
 
PDF
Spark 巨量資料處理基礎教學
byNUTC, imac
 
PDF
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
byDatabricks
 
PDF
Deep Dive Into Catalyst: Apache Spark 2.0’s Optimizer
byDatabricks
 
PDF
Introduction to Apache Spark 2.0
byKnoldus Inc.
 
PPT
Scala and spark
byFabio Fumarola
 
PPTX
Robust and Scalable ETL over Cloud Storage with Apache Spark
byDatabricks
 
PDF
使用 CLI 管理 OpenStack 平台
byNUTC, imac
 
PPTX
Meetup Spark 2.0
byJosé Carlos García Serrano
 
PDF
Docker 基本概念與指令操作
byNUTC, imac
 
PDF
London Spark Meetup Project Tungsten Oct 12 2015
byChris Fregly
 
PDF
Spark Solution for Rank Product
byMahmoud Parsian
 
PDF
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
byChris Fregly
 
PPTX
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
byRobert "Chip" Senkbeil
 
PDF
Java BigData Full Stack Development (version 2.0)
byAlexey Zinoviev
 
PPTX
Use of spark for proteomic scoring seattle presentation
bylordjoe
 
PDF
Apache Spark Essentials
bySreekanth Kodeboyena
 
PDF
Performance in Spark 2.0, PDX Spark Meetup 8/18/16
bypdx_spark
 
Spark Summit EU talk by Ted Malaska
bySpark Summit
 
Keeping Spark on Track: Productionizing Spark for ETL
byDatabricks
 
Data Source API in Spark
byDatabricks
 
Spark 巨量資料處理基礎教學
byNUTC, imac
 
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
byDatabricks
 
Deep Dive Into Catalyst: Apache Spark 2.0’s Optimizer
byDatabricks
 
Introduction to Apache Spark 2.0
byKnoldus Inc.
 
Scala and spark
byFabio Fumarola
 
Robust and Scalable ETL over Cloud Storage with Apache Spark
byDatabricks
 
使用 CLI 管理 OpenStack 平台
byNUTC, imac
 
Meetup Spark 2.0
byJosé Carlos García Serrano
 
Docker 基本概念與指令操作
byNUTC, imac
 
London Spark Meetup Project Tungsten Oct 12 2015
byChris Fregly
 
Spark Solution for Rank Product
byMahmoud Parsian
 
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
byChris Fregly
 
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
byRobert "Chip" Senkbeil
 
Java BigData Full Stack Development (version 2.0)
byAlexey Zinoviev
 
Use of spark for proteomic scoring seattle presentation
bylordjoe
 
Apache Spark Essentials
bySreekanth Kodeboyena
 
Performance in Spark 2.0, PDX Spark Meetup 8/18/16
bypdx_spark
 

Similar to Joker'16 Spark 2 (API changes; Structured Streaming; Encoders)

PDF
Introduction to Apache Spark
byAnastasios Skarlatidis
 
PDF
Apache Spark, the Next Generation Cluster Computing
byGerger
 
PDF
Introduction to Apache Spark
byVincent Poncet
 
PPTX
Spark Study Notes
byRichard Kuo
 
PDF
Apache Spark Tutorial
byFarzad Nozarian
 
ODP
Introduction to Spark with Scala
byHimanshu Gupta
 
PPTX
Spark by Adform Research, Paulius
byVasil Remeniuk
 
PPTX
Modern technologies in data science
byChucheng Hsieh
 
PPTX
Unlocking Your Hadoop Data with Apache Spark and CDH5
bySAP Concur
 
PDF
Spark浅谈
byJiahua Zhu
 
PDF
What’s New in Spark 2.0: Structured Streaming and Datasets - StampedeCon 2016
byStampedeCon
 
PDF
Osd ctw spark
byWisely chen
 
PDF
Apache Spark
byUwe Printz
 
PDF
xPatterns on Spark, Tachyon and Mesos - Bucharest meetup
byRadu Chilom
 
PDF
OCF.tw's talk about "Introduction to spark"
byGiivee The
 
PDF
Scala and big data in ICM. Scoobie, Scalding, Spark, Stratosphere. Scalar 2014
byMichał Oniszczuk
 
PDF
Introduction to Spark
byLi Ming Tsai
 
PPTX
Apache spark sneha challa- google pittsburgh-aug 25th
bySneha Challa
 
PDF
Spark手把手:[e2-spk-s02]
byErhwen Kuo
 
PDF
Streaming & Scaling Spark - London Spark Meetup 2016
byHolden Karau
 
Introduction to Apache Spark
byAnastasios Skarlatidis
 
Apache Spark, the Next Generation Cluster Computing
byGerger
 
Introduction to Apache Spark
byVincent Poncet
 
Spark Study Notes
byRichard Kuo
 
Apache Spark Tutorial
byFarzad Nozarian
 
Introduction to Spark with Scala
byHimanshu Gupta
 
Spark by Adform Research, Paulius
byVasil Remeniuk
 
Modern technologies in data science
byChucheng Hsieh
 
Unlocking Your Hadoop Data with Apache Spark and CDH5
bySAP Concur
 
Spark浅谈
byJiahua Zhu
 
What’s New in Spark 2.0: Structured Streaming and Datasets - StampedeCon 2016
byStampedeCon
 
Osd ctw spark
byWisely chen
 
Apache Spark
byUwe Printz
 
xPatterns on Spark, Tachyon and Mesos - Bucharest meetup
byRadu Chilom
 
OCF.tw's talk about "Introduction to spark"
byGiivee The
 
Scala and big data in ICM. Scoobie, Scalding, Spark, Stratosphere. Scalar 2014
byMichał Oniszczuk
 
Introduction to Spark
byLi Ming Tsai
 
Apache spark sneha challa- google pittsburgh-aug 25th
bySneha Challa
 
Spark手把手:[e2-spk-s02]
byErhwen Kuo
 
Streaming & Scaling Spark - London Spark Meetup 2016
byHolden Karau
 

More from Alexey Zinoviev

PDF
First steps in Data Mining Kindergarten
byAlexey Zinoviev
 
PDF
Выбор NoSQL базы данных для вашего проекта: "Не в свои сани не садись"
byAlexey Zinoviev
 
PDF
"Говнокод-шоу"
byAlexey Zinoviev
 
PDF
Hadoop Jungle
byAlexey Zinoviev
 
PDF
JavaDayKiev'15 Java in production for Data Mining Research projects
byAlexey Zinoviev
 
PDF
Joker'15 Java straitjackets for MongoDB
byAlexey Zinoviev
 
PDF
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)
byAlexey Zinoviev
 
PDF
JPoint'15 Mom, I so wish Hibernate for my NoSQL database...
byAlexey Zinoviev
 
PDF
HappyDev'15 Keynote: Когда все данные станут большими...
byAlexey Zinoviev
 
PDF
Мастер-класс по BigData Tools для HappyDev'15
byAlexey Zinoviev
 
PDF
Big data algorithms and data structures for large scale graphs
byAlexey Zinoviev
 
PDF
Joker'14 Java as a fundamental working tool of the Data Scientist
byAlexey Zinoviev
 
PDF
Python's slippy path and Tao of thick Pandas: give my data, Rrrrr...
byAlexey Zinoviev
 
PDF
Алгоритмы и структуры данных BigData для графов большой размерности
byAlexey Zinoviev
 
PDF
Android Geo Apps in Soviet Russia: Latitude and longitude find you
byAlexey Zinoviev
 
PDF
GDG Devfest Omsk 2013. Year of events!
byAlexey Zinoviev
 
PDF
Keynote on JavaDay Omsk 2014 about new features in Java 8
byAlexey Zinoviev
 
PDF
ALMADA 2013 (computer science school by Yandex and Microsoft Research)
byAlexey Zinoviev
 
PDF
How to port JavaScript library to Android and iOS
byAlexey Zinoviev
 
PDF
EST: Smart rate (Effective recommendation system for Taxi drivers based on th...
byAlexey Zinoviev
 
First steps in Data Mining Kindergarten
byAlexey Zinoviev
 
Выбор NoSQL базы данных для вашего проекта: "Не в свои сани не садись"
byAlexey Zinoviev
 
"Говнокод-шоу"
byAlexey Zinoviev
 
Hadoop Jungle
byAlexey Zinoviev
 
JavaDayKiev'15 Java in production for Data Mining Research projects
byAlexey Zinoviev
 
Joker'15 Java straitjackets for MongoDB
byAlexey Zinoviev
 
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)
byAlexey Zinoviev
 
JPoint'15 Mom, I so wish Hibernate for my NoSQL database...
byAlexey Zinoviev
 
HappyDev'15 Keynote: Когда все данные станут большими...
byAlexey Zinoviev
 
Мастер-класс по BigData Tools для HappyDev'15
byAlexey Zinoviev
 
Big data algorithms and data structures for large scale graphs
byAlexey Zinoviev
 
Joker'14 Java as a fundamental working tool of the Data Scientist
byAlexey Zinoviev
 
Python's slippy path and Tao of thick Pandas: give my data, Rrrrr...
byAlexey Zinoviev
 
Алгоритмы и структуры данных BigData для графов большой размерности
byAlexey Zinoviev
 
Android Geo Apps in Soviet Russia: Latitude and longitude find you
byAlexey Zinoviev
 
GDG Devfest Omsk 2013. Year of events!
byAlexey Zinoviev
 
Keynote on JavaDay Omsk 2014 about new features in Java 8
byAlexey Zinoviev
 
ALMADA 2013 (computer science school by Yandex and Microsoft Research)
byAlexey Zinoviev
 
How to port JavaScript library to Android and iOS
byAlexey Zinoviev
 
EST: Smart rate (Effective recommendation system for Taxi drivers based on th...
byAlexey Zinoviev
 

Recently uploaded

PDF
ACI 318-2205_American Concrete Institute.pdf
byericaguilerasoto1
 
PDF
Albert Pintoy - Specializing In Low-Latency
byAlbert Pintoy
 
PPTX
Unit 1 Introduction to Information Technology in Business.pptx
bygoitranjan001
 
PPTX
UnrealGameplayAbilitySystemPresentation.pptx
byBenHowenstein
 
PPTX
Emerging Trends and Research Frontiers in Chemical Engineering for Green and ...
byChemical Engineering Dept. NIT Rourkela-769008, Odisha, India
 
PPTX
Best CMMS for IoT Integration: Real-Time Asset Intelligence & Smart Maintenan...
byMaintWiz Technologies Private Limited
 
PPTX
Natural Gas fundamentals and GRU for associated gas trap.pptx
byengineerhassan
 
PDF
Lecture -06-Hybrid Policies - Chapter 7- Weeks 6-7.pdf
bysalaarmike
 
PPT
Software Engineering Unit-1 presentation for students
bykumari157687
 
PPTX
firewall Selection in production life pptx
byssuserb1479b
 
PPTX
TPM Metrics & Measurement: Drive Performance Excellence with TPM | MaintWiz
byMaintWiz Technologies Private Limited
 
PPTX
The Importance of Maintenance Budgets — Maximize Reliability & Control Costs ...
byMaintWiz Technologies Private Limited
 
PDF
AI-Driven CTI for Business: Emerging Threats, Attack Strategies, and Defensiv...
byAIRCC Publishing Corporation
 
PPTX
ISO 14224 Compliance & CMMS Software — A Comprehensive Guide for Reliable Mai...
byMaintWiz Technologies Private Limited
 
PPTX
Takt Time vs Cycle Time vs Lead Time.pptx
byE Concepts
 
PPTX
22304_BCO_CO3_LO4_PPT MSBTE Building construction.pptx
byPrasadBahekar4
 
PPTX
Plant Performance Strategies: Enhanced Reliability & Operational Efficiency w...
byMaintWiz Technologies Private Limited
 
PPTX
Introduction Blockchains and Smart Contracts
bybobinson
 
PDF
Structural Conservation Appraisal of Indian Monuments Preserving India’s Arch...
byAman Kumar Singh
 
PPTX
Optimizing Operations: Key Elements of a Successful Plant Maintenance Plan — ...
byMaintWiz Technologies Private Limited
 
ACI 318-2205_American Concrete Institute.pdf
byericaguilerasoto1
 
Albert Pintoy - Specializing In Low-Latency
byAlbert Pintoy
 
Unit 1 Introduction to Information Technology in Business.pptx
bygoitranjan001
 
UnrealGameplayAbilitySystemPresentation.pptx
byBenHowenstein
 
Emerging Trends and Research Frontiers in Chemical Engineering for Green and ...
byChemical Engineering Dept. NIT Rourkela-769008, Odisha, India
 
Best CMMS for IoT Integration: Real-Time Asset Intelligence & Smart Maintenan...
byMaintWiz Technologies Private Limited
 
Natural Gas fundamentals and GRU for associated gas trap.pptx
byengineerhassan
 
Lecture -06-Hybrid Policies - Chapter 7- Weeks 6-7.pdf
bysalaarmike
 
Software Engineering Unit-1 presentation for students
bykumari157687
 
firewall Selection in production life pptx
byssuserb1479b
 
TPM Metrics & Measurement: Drive Performance Excellence with TPM | MaintWiz
byMaintWiz Technologies Private Limited
 
The Importance of Maintenance Budgets — Maximize Reliability & Control Costs ...
byMaintWiz Technologies Private Limited
 
AI-Driven CTI for Business: Emerging Threats, Attack Strategies, and Defensiv...
byAIRCC Publishing Corporation
 
ISO 14224 Compliance & CMMS Software — A Comprehensive Guide for Reliable Mai...
byMaintWiz Technologies Private Limited
 
Takt Time vs Cycle Time vs Lead Time.pptx
byE Concepts
 
22304_BCO_CO3_LO4_PPT MSBTE Building construction.pptx
byPrasadBahekar4
 
Plant Performance Strategies: Enhanced Reliability & Operational Efficiency w...
byMaintWiz Technologies Private Limited
 
Introduction Blockchains and Smart Contracts
bybobinson
 
Structural Conservation Appraisal of Indian Monuments Preserving India’s Arch...
byAman Kumar Singh
 
Optimizing Operations: Key Elements of a Successful Plant Maintenance Plan — ...
byMaintWiz Technologies Private Limited
 

Joker'16 Spark 2 (API changes; Structured Streaming; Encoders)

  • 1.
    Spark 2 Alexey Zinovyev,Java/BigData Trainer in EPAM
  • 2.
    About With IT since2007 With Java since 2009 With Hadoop since 2012 With EPAM since 2015
  • 3.
    3Spark 2 fromZinoviev Alexey Contacts E-mail : Alexey_Zinovyev@epam.com Twitter : @zaleslaw @BigDataRussia Facebook: https://www.facebook.com/zaleslaw vk.com/big_data_russia Big Data Russia vk.com/java_jvm Java & JVM langs
  • 4.
    4Spark 2 fromZinoviev Alexey Sprk Dvlprs! Let’s start!
  • 5.
    5Spark 2 fromZinoviev Alexey < SPARK 2.0
  • 6.
    6Spark 2 fromZinoviev Alexey Modern Java in 2016Big Data in 2014
  • 7.
    7Spark 2 fromZinoviev Alexey Big Data in 2017
  • 8.
    8Spark 2 fromZinoviev Alexey Machine Learning EVERYWHERE
  • 9.
    9Spark 2 fromZinoviev Alexey Machine Learning vs Traditional Programming
  • 10.
    10Spark 2 fromZinoviev Alexey
  • 11.
    11Spark 2 fromZinoviev Alexey Something wrong with HADOOP
  • 12.
    12Spark 2 fromZinoviev Alexey Hadoop is not SEXY
  • 13.
    13Spark 2 fromZinoviev Alexey Whaaaat?
  • 14.
    14Spark 2 fromZinoviev Alexey Map Reduce Job Writing
  • 15.
    15Spark 2 fromZinoviev Alexey MR code
  • 16.
    16Spark 2 fromZinoviev Alexey Hadoop Developers Right Now
  • 17.
    17Spark 2 fromZinoviev Alexey Iterative Calculations 10x – 100x
  • 18.
    18Spark 2 fromZinoviev Alexey MapReduce vs Spark
  • 19.
    19Spark 2 fromZinoviev Alexey MapReduce vs Spark
  • 20.
    20Spark 2 fromZinoviev Alexey MapReduce vs Spark
  • 21.
    21Spark 2 fromZinoviev Alexey SPARK INTRO
  • 22.
    22Spark 2 fromZinoviev Alexey Loading val localData = (5,7,1,12,10,25) val ourFirstRDD = sc.parallelize(localData) val textFile = sc.textFile("hdfs://...")
  • 23.
    23Spark 2 fromZinoviev Alexey Loading val localData = (5,7,1,12,10,25) val ourFirstRDD = sc.parallelize(localData) val textFile = sc.textFile("hdfs://...")
  • 24.
    24Spark 2 fromZinoviev Alexey Loading val localData = (5,7,1,12,10,25) val ourFirstRDD = sc.parallelize(localData) // from file val textFile = sc.textFile("hdfs://...")
  • 25.
    25Spark 2 fromZinoviev Alexey Loading val localData = (5,7,1,12,10,25) val ourFirstRDD = sc.parallelize(localData) // from file val textFile = sc.textFile("hdfs://...")
  • 26.
    26Spark 2 fromZinoviev Alexey Loading val localData = (5,7,1,12,10,25) val ourFirstRDD = sc.parallelize(localData) // from file val textFile = sc.textFile("hdfs://...")
  • 27.
    27Spark 2 fromZinoviev Alexey Word Count val textFile = sc.textFile("hdfs://...") val counts = textFile .flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs://...")
  • 28.
    28Spark 2 fromZinoviev Alexey Word Count val textFile = sc.textFile("hdfs://...") val counts = textFile .flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs://...")
  • 29.
    29Spark 2 fromZinoviev Alexey Word Count val textFile = sc.textFile("hdfs://...") val counts = textFile .flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs://...")
  • 30.
    30Spark 2 fromZinoviev Alexey SQL val hive = new HiveContext(spark) hive.hql(“CREATE TABLE IF NOT EXISTS src (key INT, value STRING)”) hive.hql(“LOAD DATA LOCAL INPATH ‘…/kv1.txt’ INTO TABLE src”) val results = hive.hql(“FROM src SELECT key, value”).collect()
  • 31.
    31Spark 2 fromZinoviev Alexey SQL val hive = new HiveContext(spark) hive.hql(“CREATE TABLE IF NOT EXISTS src (key INT, value STRING)”) hive.hql(“LOAD DATA LOCAL INPATH ‘…/kv1.txt’ INTO TABLE src”) val results = hive.hql(“FROM src SELECT key, value”).collect()
  • 32.
    32Spark 2 fromZinoviev Alexey SQL val hive = new HiveContext(spark) hive.hql(“CREATE TABLE IF NOT EXISTS src (key INT, value STRING)”) hive.hql(“LOAD DATA LOCAL INPATH ‘…/kv1.txt’ INTO TABLE src”) val results = hive.hql(“FROM src SELECT key, value”).collect()
  • 33.
    33Spark 2 fromZinoviev Alexey SQL val hive = new HiveContext(spark) hive.hql(“CREATE TABLE IF NOT EXISTS src (key INT, value STRING)”) hive.hql(“LOAD DATA LOCAL INPATH ‘…/kv1.txt’ INTO TABLE src”) val results = hive.hql(“FROM src SELECT key, value”).collect()
  • 34.
    34Spark 2 fromZinoviev Alexey RDD Factory
  • 35.
    35Spark 2 fromZinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
  • 36.
    36Spark 2 fromZinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
  • 37.
    37Spark 2 fromZinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
  • 38.
    38Spark 2 fromZinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
  • 39.
    39Spark 2 fromZinoviev Alexey DStream val conf = new SparkConf().setMaster("local[2]") .setAppName("NetworkWordCount") val ssc = new StreamingContext(conf, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination()
  • 40.
    40Spark 2 fromZinoviev Alexey It’s very easy to understand Graph algorithms def pregel[A](initialMsg: A, maxIterations: Int, activeDirection: EdgeDirection)( vprog: (VertexID, VD, A) => VD, sendMsg: EdgeTriplet[VD, ED] => Iterator[(VertexID,A)], mergeMsg: (A, A) => A) : Graph[VD, ED] = { <Your Pregel Computations> }
  • 41.
    41Spark 2 fromZinoviev Alexey SPARK 2.0 DISCUSSION
  • 42.
    42Spark 2 fromZinoviev Alexey Spark Family
  • 43.
    43Spark 2 fromZinoviev Alexey Spark Family
  • 44.
    44Spark 2 fromZinoviev Alexey Case #0 : How to avoid DStreams with RDD-like API?
  • 45.
    45Spark 2 fromZinoviev Alexey Continuous Applications
  • 46.
    46Spark 2 fromZinoviev Alexey Continuous Applications cases • Updating data that will be served in real time • Extract, transform and load (ETL) • Creating a real-time version of an existing batch job • Online machine learning
  • 47.
    47Spark 2 fromZinoviev Alexey Write Batches
  • 48.
    48Spark 2 fromZinoviev Alexey Catch Streaming
  • 49.
    49Spark 2 fromZinoviev Alexey The main concept of Structured Streaming You can express your streaming computation the same way you would express a batch computation on static data.
  • 50.
    50Spark 2 fromZinoviev Alexey Batch // Read JSON once from S3 logsDF = spark.read.json("s3://logs") // Transform with DataFrame API and save logsDF.select("user", "url", "date") .write.parquet("s3://out")
  • 51.
    51Spark 2 fromZinoviev Alexey Real Time // Read JSON continuously from S3 logsDF = spark.readStream.json("s3://logs") // Transform with DataFrame API and save logsDF.select("user", "url", "date") .writeStream.parquet("s3://out") .start()
  • 52.
    52Spark 2 fromZinoviev Alexey Unlimited Table
  • 53.
    53Spark 2 fromZinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
  • 54.
    54Spark 2 fromZinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
  • 55.
    55Spark 2 fromZinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count()
  • 56.
    56Spark 2 fromZinoviev Alexey WordCount from Socket val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words = lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count() Don’t forget to start Streaming
  • 57.
    57Spark 2 fromZinoviev Alexey WordCount with Structured Streaming
  • 58.
    58Spark 2 fromZinoviev Alexey Structured Streaming provides … • fast & scalable • fault-tolerant • end-to-end with exactly-once semantic • stream processing • ability to use DataFrame/DataSet API for streaming
  • 59.
    59Spark 2 fromZinoviev Alexey Structured Streaming provides (in dreams) … • fast & scalable • fault-tolerant • end-to-end with exactly-once semantic • stream processing • ability to use DataFrame/DataSet API for streaming
  • 60.
    60Spark 2 fromZinoviev Alexey Let’s UNION streaming and static DataSets
  • 61.
    61Spark 2 fromZinoviev Alexey Let’s UNION streaming and static DataSets org.apache.spark.sql. AnalysisException: Union between streaming and batch DataFrames/Datasets is not supported;
  • 62.
    62Spark 2 fromZinoviev Alexey Let’s UNION streaming and static DataSets Go to UnsupportedOperationChecker.scala and check your operation
  • 63.
    63Spark 2 fromZinoviev Alexey Case #1 : We should think about optimization in RDD terms
  • 64.
    64Spark 2 fromZinoviev Alexey Single Thread collection
  • 65.
    65Spark 2 fromZinoviev Alexey No perf issues, right?
  • 66.
    66Spark 2 fromZinoviev Alexey The main concept more partitions = more parallelism
  • 67.
    67Spark 2 fromZinoviev Alexey Do it parallel
  • 68.
    68Spark 2 fromZinoviev Alexey I’d like NARROW
  • 69.
    69Spark 2 fromZinoviev Alexey Map, filter, filter
  • 70.
    70Spark 2 fromZinoviev Alexey GroupByKey, join
  • 71.
    71Spark 2 fromZinoviev Alexey Case #2 : DataFrames suggest mix SQL and Scala functions
  • 72.
    72Spark 2 fromZinoviev Alexey History of Spark APIs
  • 73.
    73Spark 2 fromZinoviev Alexey RDD rdd.filter(_.age > 21) // RDD df.filter("age > 21") // DataFrame SQL-style df.filter(df.col("age").gt(21)) // Expression style dataset.filter(_.age < 21); // Dataset API
  • 74.
    74Spark 2 fromZinoviev Alexey History of Spark APIs
  • 75.
    75Spark 2 fromZinoviev Alexey SQL rdd.filter(_.age > 21) // RDD df.filter("age > 21") // DataFrame SQL-style df.filter(df.col("age").gt(21)) // Expression style dataset.filter(_.age < 21); // Dataset API
  • 76.
    76Spark 2 fromZinoviev Alexey Expression rdd.filter(_.age > 21) // RDD df.filter("age > 21") // DataFrame SQL-style df.filter(df.col("age").gt(21)) // Expression style dataset.filter(_.age < 21); // Dataset API
  • 77.
    77Spark 2 fromZinoviev Alexey History of Spark APIs
  • 78.
    78Spark 2 fromZinoviev Alexey DataSet rdd.filter(_.age > 21) // RDD df.filter("age > 21") // DataFrame SQL-style df.filter(df.col("age").gt(21)) // Expression style dataset.filter(_.age < 21); // Dataset API
  • 79.
    79Spark 2 fromZinoviev Alexey Case #2 : DataFrame is referring to data attributes by name
  • 80.
    80Spark 2 fromZinoviev Alexey DataSet = RDD’s types + DataFrame’s Catalyst • RDD API • compile-time type-safety • off-heap storage mechanism • performance benefits of the Catalyst query optimizer • Tungsten
  • 81.
    81Spark 2 fromZinoviev Alexey DataSet = RDD’s types + DataFrame’s Catalyst • RDD API • compile-time type-safety • off-heap storage mechanism • performance benefits of the Catalyst query optimizer • Tungsten
  • 82.
    82Spark 2 fromZinoviev Alexey Structured APIs in SPARK
  • 83.
    83Spark 2 fromZinoviev Alexey Unified API in Spark 2.0 DataFrame = Dataset[Row] Dataframe is a schemaless (untyped) Dataset now
  • 84.
    84Spark 2 fromZinoviev Alexey Define case class case class User(email: String, footSize: Long, name: String) // DataFrame -> DataSet with Users val userDS = spark.read.json("/home/tmp/datasets/users.json").as[User] userDS.map(_.name).collect() userDS.filter(_.footSize > 38).collect() ds.rdd // IF YOU REALLY WANT
  • 85.
    85Spark 2 fromZinoviev Alexey Read JSON case class User(email: String, footSize: Long, name: String) // DataFrame -> DataSet with Users val userDS = spark.read.json("/home/tmp/datasets/users.json").as[User] userDS.map(_.name).collect() userDS.filter(_.footSize > 38).collect() ds.rdd // IF YOU REALLY WANT
  • 86.
    86Spark 2 fromZinoviev Alexey Filter by Field case class User(email: String, footSize: Long, name: String) // DataFrame -> DataSet with Users val userDS = spark.read.json("/home/tmp/datasets/users.json").as[User] userDS.map(_.name).collect() userDS.filter(_.footSize > 38).collect() ds.rdd // IF YOU REALLY WANT
  • 87.
    87Spark 2 fromZinoviev Alexey Case #3 : Spark has many contexts
  • 88.
    88Spark 2 fromZinoviev Alexey Spark Session • New entry point in spark for creating datasets • Replaces SQLContext, HiveContext and StreamingContext • Move from SparkContext to SparkSession signifies move away from RDD
  • 89.
    89Spark 2 fromZinoviev Alexey Spark Session val sparkSession = SparkSession.builder .master("local") .appName("spark session example") .getOrCreate() val df = sparkSession.read .option("header","true") .csv("src/main/resources/names.csv") df.show()
  • 90.
    90Spark 2 fromZinoviev Alexey No, I want to create my lovely RDDs
  • 91.
    91Spark 2 fromZinoviev Alexey Where’s parallelize() method?
  • 92.
    92Spark 2 fromZinoviev Alexey RDD? case class User(email: String, footSize: Long, name: String) // DataFrame -> DataSet with Users val userDS = spark.read.json("/home/tmp/datasets/users.json").as[User] userDS.map(_.name).collect() userDS.filter(_.footSize > 38).collect() ds.rdd // IF YOU REALLY WANT
  • 93.
    93Spark 2 fromZinoviev Alexey Case #4 : Spark uses Java serialization A LOT
  • 94.
    94Spark 2 fromZinoviev Alexey Two choices to distribute data across cluster • Java serialization By default with ObjectOutputStream • Kryo serialization Should register classes (no support of Serialazible)
  • 95.
    95Spark 2 fromZinoviev Alexey The main problem: overhead of serializing Each serialized object contains the class structure as well as the values
  • 96.
    96Spark 2 fromZinoviev Alexey The main problem: overhead of serializing Each serialized object contains the class structure as well as the values Don’t forget about GC
  • 97.
    97Spark 2 fromZinoviev Alexey Tungsten Compact Encoding
  • 98.
    98Spark 2 fromZinoviev Alexey Encoder’s concept Generate bytecode to interact with off-heap & Give access to attributes without ser/deser
  • 99.
    99Spark 2 fromZinoviev Alexey Encoders
  • 100.
    100Spark 2 fromZinoviev Alexey No custom encoders
  • 101.
    101Spark 2 fromZinoviev Alexey Case #5 : Not enough storage levels 
  • 102.
    102Spark 2 fromZinoviev Alexey Caching in Spark • Frequently used RDD can be stored in memory • One method, one short-cut: persist(), cache() • SparkContext keeps track of cached RDD • Serialized or deserialized Java objects
  • 103.
    103Spark 2 fromZinoviev Alexey Full list of options • MEMORY_ONLY • MEMORY_AND_DISK • MEMORY_ONLY_SER • MEMORY_AND_DISK_SER • DISK_ONLY • MEMORY_ONLY_2, MEMORY_AND_DISK_2
  • 104.
    104Spark 2 fromZinoviev Alexey Spark Core Storage Level • MEMORY_ONLY (default for Spark Core) • MEMORY_AND_DISK • MEMORY_ONLY_SER • MEMORY_AND_DISK_SER • DISK_ONLY • MEMORY_ONLY_2, MEMORY_AND_DISK_2
  • 105.
    105Spark 2 fromZinoviev Alexey Spark Streaming Storage Level • MEMORY_ONLY (default for Spark Core) • MEMORY_AND_DISK • MEMORY_ONLY_SER (default for Spark Streaming) • MEMORY_AND_DISK_SER • DISK_ONLY • MEMORY_ONLY_2, MEMORY_AND_DISK_2
  • 106.
    106Spark 2 fromZinoviev Alexey Developer API to make new Storage Levels
  • 107.
    107Spark 2 fromZinoviev Alexey What’s the most popular file format in BigData?
  • 108.
    108Spark 2 fromZinoviev Alexey Case #6 : External libraries to read CSV
  • 109.
    109Spark 2 fromZinoviev Alexey Easy to read CSV data = sqlContext.read.format("csv") .option("header", "true") .option("inferSchema", "true") .load("/datasets/samples/users.csv") data.cache() data.createOrReplaceTempView(“users") display(data)
  • 110.
    110Spark 2 fromZinoviev Alexey Case #7 : How to measure Spark performance?
  • 111.
    111Spark 2 fromZinoviev Alexey You'd measure performance!
  • 112.
    112Spark 2 fromZinoviev Alexey TPCDS 99 Queries http://bit.ly/2dObMsH
  • 113.
    113Spark 2 fromZinoviev Alexey
  • 114.
    114Spark 2 fromZinoviev Alexey How to benchmark Spark
  • 115.
    115Spark 2 fromZinoviev Alexey Special Tool from Databricks Benchmark Tool for SparkSQL https://github.com/databricks/spark-sql-perf
  • 116.
    116Spark 2 fromZinoviev Alexey Spark 2 vs Spark 1.6
  • 117.
    117Spark 2 fromZinoviev Alexey Case #8 : What’s faster: SQL or DataSet API?
  • 118.
    118Spark 2 fromZinoviev Alexey Job Stages in old Spark
  • 119.
    119Spark 2 fromZinoviev Alexey Scheduler Optimizations
  • 120.
    120Spark 2 fromZinoviev Alexey Catalyst Optimizer for DataFrames
  • 121.
    121Spark 2 fromZinoviev Alexey Unified Logical Plan DataFrame = Dataset[Row]
  • 122.
    122Spark 2 fromZinoviev Alexey Bytecode
  • 123.
    123Spark 2 fromZinoviev Alexey DataSet.explain() == Physical Plan == Project [avg(price)#43,carat#45] +- SortMergeJoin [color#21], [color#47] :- Sort [color#21 ASC], false, 0 : +- TungstenExchange hashpartitioning(color#21,200), None : +- Project [avg(price)#43,color#21] : +- TungstenAggregate(key=[cut#20,color#21], functions=[(avg(cast(price#25 as bigint)),mode=Final,isDistinct=false)], output=[color#21,avg(price)#43]) : +- TungstenExchange hashpartitioning(cut#20,color#21,200), None : +- TungstenAggregate(key=[cut#20,color#21], functions=[(avg(cast(price#25 as bigint)),mode=Partial,isDistinct=false)], output=[cut#20,color#21,sum#58,count#59L]) : +- Scan CsvRelation(-----) +- Sort [color#47 ASC], false, 0 +- TungstenExchange hashpartitioning(color#47,200), None +- ConvertToUnsafe +- Scan CsvRelation(----)
  • 124.
    124Spark 2 fromZinoviev Alexey Case #9 : Why does explain() show so many Tungsten things?
  • 125.
    125Spark 2 fromZinoviev Alexey How to be effective with CPU • Runtime code generation • Exploiting cache locality • Off-heap memory management
  • 126.
    126Spark 2 fromZinoviev Alexey Tungsten’s goal Push performance closer to the limits of modern hardware
  • 127.
    127Spark 2 fromZinoviev Alexey Maybe something UNSAFE?
  • 128.
    128Spark 2 fromZinoviev Alexey UnsafeRowFormat • Bit set for tracking null values • Small values are inlined • For variable-length values are stored relative offset into the variablelength data section • Rows are always 8-byte word aligned • Equality comparison and hashing can be performed on raw bytes without requiring additional interpretation
  • 129.
    129Spark 2 fromZinoviev Alexey Case #10 : Can I influence on Memory Management in Spark?
  • 130.
    130Spark 2 fromZinoviev Alexey Case #11 : Should I tune generation’s stuff?
  • 131.
    131Spark 2 fromZinoviev Alexey Cached Data
  • 132.
    132Spark 2 fromZinoviev Alexey During operations
  • 133.
    133Spark 2 fromZinoviev Alexey For your needs
  • 134.
    134Spark 2 fromZinoviev Alexey For Dark Lord
  • 135.
    135Spark 2 fromZinoviev Alexey IN CONCLUSION
  • 136.
    136Spark 2 fromZinoviev Alexey We have no ability… • join structured streaming and other sources to handle it • one unified ML API • GraphX rethinking and redesign • Custom encoders • Datasets everywhere • integrate with something important
  • 137.
    137Spark 2 fromZinoviev Alexey Roadmap • Support other data sources (not only S3 + HDFS) • Transactional updates • Dataset is one DSL for all operations • GraphFrames + Structured MLLib • Tungsten: custom encoders • The RDD-based API is expected to be removed in Spark 3.0
  • 138.
    138Spark 2 fromZinoviev Alexey And we can DO IT! Real-Time Data-Marts Batch Data-Marts Relations Graph Ontology Metadata Search Index Events & Alarms Real-time Dashboarding Events & Alarms All Raw Data backup is stored here Real-time Data Ingestion Batch Data Ingestion Real-Time ETL & CEP Batch ETL & Raw Area Scheduler Internal External Social HDFS → CFS as an option Time-Series Data Titan & KairosDB store data in Cassandra Push Events & Alarms (Email, SNMP etc.)
  • 139.
    139Spark 2 fromZinoviev Alexey First Part
  • 140.
    140Spark 2 fromZinoviev Alexey Second Part
  • 141.
    141Spark 2 fromZinoviev Alexey Contacts E-mail : Alexey_Zinovyev@epam.com Twitter : @zaleslaw @BigDataRussia Facebook: https://www.facebook.com/zaleslaw vk.com/big_data_russia Big Data Russia vk.com/java_jvm Java & JVM langs
  • 142.
    142Spark 2 fromZinoviev Alexey Any questions?