Databricks, profile picture
Uploaded byDatabricks
2,887 views

Demystifying DataFrame and Dataset with Kazuaki Ishizaki

Kazuaki Ishizaki from IBM Research presents enhancements in Apache Spark 2.2, highlighting the performance improvements of Datasets over DataFrames and RDDs. The document details benchmarking results, showing that Datasets have become significantly faster due to optimizations in the Catalyst optimizer and code generation. Key discussions include execution differences between Datasets and DataFrames, performance bottlenecks in earlier Spark versions, and the implications for application programmers and developers in machine learning pipelines.

Data & Analytics◦
Related topics:
Apache Spark•

Embed presentation

Downloaded 151 times
Kazuaki Ishizaki IBM Research – Tokyo @kiszk Demystifying DataFrame and Dataset #SFdev20
About Me – Kazuaki Ishizaki • Researcher at IBM Research in compiler optimizations • Working for IBM Java virtual machine over 20 years – In particular, just-in-time compiler • Contributor to SQL package in Apache Spark • Committer of GPUEnabler – Apache Spark Plug-in to execute GPU code on Spark • https://github.com/IBMSparkGPU/GPUEnabler • Homepage: http://ibm.biz/ishizaki • Github: https://github.com/kiszk, Twitter: @kiszk Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki2
Spark 2.2 makes Dataset Faster Compared to Spark 2.0 (also 2.1) • 1.6x faster for map() with scalar variable • 4.5x faster for map() with primitive array Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki3 ds = (1 to ...).toDS.cache ds.map(value => value + 1) ds = Seq(Array(…), Array(…), …).toDS.cache ds.map(array => Array(array(0))) from enhancements in Catalyst optimizer and codegen
How Does It Matter? • Application programmers – ML pipeline will become faster • Library developers – You will want to use Dataset instead of RDD • SQL package developers – You will know detail on Dataset Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki4
DataFrame, Dataset, and RDD df = (1 to 100).toDF.cache df.selectExpr(“value + 1”) ds = (1 to 100).toDS.cache ds.map(value => value + 1) SQLDataFrame (DF) Dataset (DS) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki5 RDD rdd = sc.parallelize( (1 to 100)).cache rdd.map(value => value + 1) Based on lambda expressionBased on SQL operation
How DF, DS, and RDD Work • We expect the same performance on DF and DS From Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Michael Armbrust DS DF Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki6 Catalyst // generated Java code // for DF and DS while (itr.hasNext()) { Row r = (Row)itr.next(); int v = r.getInt(0); … }
Dataset for Int Value is Slow on Spark 2.0 • 1.7x slow for addition to scalar int value 0 0.5 1 1.5 2 Relative execution time over DataFrame DataFrame Dataset df.selectExpr(“value + 1”) ds.map(value => value + 1) All experiments are done using 16-core Intel E2667-v3 3.2GHz, with 384GB mem, Ubuntu 16.04, OpenJDK 1.8.0u212-b13 with 256GB heap, Spark master=local[1] Spark branch-2.0 and branch-2.2, ds/df/rdd are cached Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki7 Shorter is better
Dataset for Int Array is Extremely Slow on Spark 2.0 • 54x slow for creating a new array for primitive int array 0 10 20 30 40 50 60 Relative execution time over DataFrame DataFrame Dataset df = Seq(Array(…), Array(…), …) .toDF(“a”).cache df.selectExpr(“Array(a[0])”) ds = Seq(Array(…), Array(…), …) .toDS.cache ds.map(a => Array(a(0))) SPARK-16070 pointed out this performance problem Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki8 Shorter is better
Outline • How DataFrame and Dataset are executed? • Deep dive for two cases – Why Dataset was slow – How Spark 2.2 improves performance • Deep dive for another case – Why Dataset is slow – How future Spark will improve performance • How Dataset is used in ML pipeline? • Next Steps Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki9
Simple Generated Code for int with DF • Generated code performs `+` to int – Use strongly-typed ‘int’ variable 1: while (itr.hasNext()) { // execute a row 2: // get a value from a row in DF 3: int value =((Row)itr.next()).getInt(0); 4: // compute a new value 5: int mapValue = value + 1; 6: // store a new value to a row in DF 7: outRow.write(0, mapValue); 8: append(outRow); 9: } // df: DataFrame for int … df.selectExpr(“value + 1”) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki10 Catalyst generates Java code Catalyst
Weird Generated Code with DS • Generated code performs four data conversions – Use standard API with generic type method apply(Object) while (itr.hasNext()) { int value = ((Row)itr.next()).getInt(0); Object objValue = new Integer(value); int mapValue = ((Integer) mapFunc.apply(objVal)).toValue(); outRow.write(0, mapValue); append(outRow); } Object apply(Object obj) { int value = Integer(obj).toValue(); return new Integer(apply$II(value)); } int apply$II(int value) { return value + 1; } // ds: Dataset[Int] … ds.map(value => value + 1) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki11 Catalyst generates Java code Scalac generates these functions from lambda expression Catalyst
Simple Generated Code with DS on Spark 2.2 • SPARK-19008 enables generated code to use int value – Use non-standard API with type-specialized method apply$II(int) while (itr.hasNext()) { int value = ((Row)itr.next()).getInt(0); int mapValue = mapFunc.apply$II(value); outRow.write(0, mapValue); append(outRow); } int apply$II(int value) { return value + 1;} // ds: Dataset[Int] … ds.map(value => value + 1) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki12 Catalyst Scalac generates this function from lambda expression
Dataset is Not Slow on Spark 2.2 0 0.5 1 1.5 2 Relative execution time over DataFrame DataFrame Dataset df.selectExpr(“value + 1”) ds.map(value => value + 1) Shorter is better Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki13 1.6x • Improve performance by 1.6x compared to Spark 2.0 – DataFrame is 9% faster than Dataset
RDD/DF/DS achieve close performance on Spark 2.2 • RDD is 5% faster than DataFrame 0 0.5 1 1.5 2 Relative execution time over DataFrame RDD DataFrame Dataset df.selectExpr(“value + 1”) ds.map(value => value + 1) rdd.map(value => value + 1) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki14 9% 5% Shorter is better
Simple Generated Code for Array with DF • All operations access data in internal data format for array ArrayData (devised in Project Tungsten) // df: DataFrame for Array(Int) … df.selectExpr(“Array(a[0])”) ArrayData inArray, outArray; while (itr.hasNext()) { inArray = ((Row)itr.next()).getArray(0); int value = inArray.getInt(0)); //a[0] outArray = new UnsafeArrayData(); ... outArray.setInt(0, value); //Array(a[0]) outArray.writeToMemory(outRow); append(outRow); } Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki15 Catalyst generates Java code Internal data format (ArrayData) inArray.getInt(0) outArray.setInt(0, …) Catalyst
Lambda Expression with DS requires Java Object • Need serialization/deserialzation (Ser/De) between internal data format and Java object for lambda expression // ds: DataSet[Array(Int)] … ds.map(a => Array(a(0))) ArrayData inArray, outArray; while (itr.hasNext()) { inArray = ((Row)itr.next()).getArray(0); append(outRow); } Internal data formatJava object Ser/De Array(a(0)) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki16 int[] mapArray = Array(a(0)); Java bytecode for lambda expression Catalyst Deserialization Serialization Ser/De
ArrayData inArray, outArray; while (itr.hasNext()) { inArray = ((Row)itr.next().getArray(0); Object[] tmp = new Object[inArray.numElements()]; for (int i = 0; i < tmp.length; i ++) { tmp[i] = (inArray.isNullAt(i)) ? Null : inArray.getInt(i); } ArrayData array = new GenericIntArrayData(tmpArray); int[] javaArray = array.toIntArray(); int[] mapArray = (int[])map_func.apply(javaArray); outArray = new GenericArrayData(mapArray); for (int i = 0; i < outArray.numElements(); i++) { if (outArray.isNullAt(i)) { arrayWriter.setNullInt(i); } else { arrayWriter.write(i, outArray.getInt(i)); } } append(outRow); } ArrayData inArray, outArray; while (itr.hasNext()) { inArray = ((Row)itr.next().getArray(0); append(outRow); } Extremely Weird Code for Array with DS on Spark 2.0 • Data conversion is too slow • Element-wise copy is slow // ds: DataSet[Array(Int)] … ds.map(a => Array(a(0))) Data conversion Element-wise data copy Element-wise data copy Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki17 Data conversion Copy with Java object creation Element-wise data copy Copy each element with null check int[] mapArray = Array(a(0)); Catalyst Data conversion Element-wise data copy Ser De
Simple Generated Code for Array on Spark 2.2 • SPARK-15985, SPARK-17490, and SPARK-15962 simplify Ser/De by using bulk data copy – Data conversion and element-wise copy are not used – Bulk copy is faster // ds: DataSet[Array(Int)] … ds.map(a => Array(a(0))) while (itr.hasNext()) { inArray =((Row)itr.next()).getArray(0); int[] javaArray = inArray.toIntArray(); int[] mapArray = (int[])mapFunc.apply(javaArray); outArray = UnsafeArrayData .fromPrimitiveArray(mapArray); outArray.writeToMemory(outRow); append(outRow); } Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki18 Bulk data copy Copy whole array using memcpy() Catalyst while (itr.hasNext()) { inArray =((Row)itr.next()).getArray(0); int[]mapArray=(int[])map.apply(javaArray); append(outRow); } Bulk data copy int[] mapArray = Array(a(0)); Bulk data copy Bulk data copy
0 10 20 30 40 50 60 Relative execution time over DataFrame DataFrame Dataset Dataset for Array is Not Extremely Slow over DataFrame • Improve performance by 4.5 compared to Spark 2.0 – DataFrame is 12x faster than Dataset ds = Seq(Array(…), Array(…), …) .toDS.cache ds.map(a => Array(a(0))) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki19 4.5x df = Seq(Array(…), Array(…), …) .toDF（”a”).cache df.selectExpr(“Array(a[0])”) Would this overhead be negligible if map() is much computation intensive? 12x Shorter is better
RDD and DataFrame for Array Achieve Close Performance • DataFrame is 1.2x faster than RDD 0 5 10 15 Relative execution time over DataFrame RDD DataFrame Dataset ds = Seq(Array(…), Array(…), …) .toDS.cache ds.map(a => Array(a(0))) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki20 df = Seq(Array(…), Array(…), …) .toDF（”a”).cache df.selectExpr(“Array(a[0])”) rdd = sc.parallelize(Seq( Array(…), Array(…), …)).cache rdd.map(a => Array(a(0))) 1.2 12 Shorter is better
Enable Lambda Expression to Use Internal Data Format • Our prototype modifies Java bytecode of lambda expression to access internal data format – We improved performance by avoiding Ser/De // ds: DataSet[Array(Int)] … ds.map(a => Array(a(0))) while (itr.hasNext()) { inArray = ((Row)itr.next().getArray(0); outArray = mapFunc.applyTungsten(inArray); outArray.writeToMemory(outRow); append(outRow); } Internal data format “Accelerating Spark Datasets by inlining deserialization”, our academic paper at IPDPS2017 1 1 apply(Object) 1011 applyTungsten(ArrayData) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki21 Catalyst generates Java code Catalyst Our prototype
Outline • How DataFrame and Dataset are executed? • Deep dive for two cases – Why Dataset is slow – How Spark 2.2 improve performance • Deep dive for another case – Why Dataset is slow • How Dataset are used in ML pipeline? • Next Steps Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki22
0 0.5 1 1.5 Relative execution time over DataFrame RDD DataFrame Dataset Dataset for Two Scalar Adds is Slow on Spark 2.2 • DataFrame and Dataset are faster than RDD • Dataset is 1.1x slower than DataFrame df.selectExpr(“value + 1”) .selectExpr(“value + 2”) ds.map(value => value + 1) .map(value => value + 2) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki23 rdd.map(value => value + 1) .map(value => value + 2) Shorter is better ☺  1.1 1.4
Adds are Combined with DF • Spark 2.2 understands operations in selectExpr() and combines two adds into one add ‘value + 3’ while (itr.hasNext()) { Row row = (Row)itr.next(); int value = row.getInt(0); int mapValue = value + 3; projectRow.write(0, mapValue); append(projectRow); } // df: DataFrame for Int … df.selectExpr(“value + 1”) .selectExpr(“value + 2”) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki24 Catalyst Catalyst generates Java code
Two Method Calls for Adds Remain with DS • Spark 2.2 cannot understand lambda expressions stored as Java bytecode while (itr.hasNext()) { int value = ((Row)itr.next()).getInt(0); int mapValue = mapFunc1.apply$II(value); mapValue += mapFunc2.apply$II(mapValue); outRow.write(0, mapValue); append(outRow); } // ds: Dataset[Int] … ds.map(value => value + 1) .map(value => value + 2) class MapFunc1 { int apply$II(int value) { return value + 1;}} class MapFunc2 { int apply$II(int value) { return value + 2;}} Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki25 Catalyst Scalac generates these functions from lambda expression
class MapFunc1 { int apply$II(int value) { return value + 1;}} class MapFunc2 { int apply$II(int value) { return value + 2;}} Adds Will be Combined on Future Spark 2.x • SPARK-14083 will allow future Spark to understand Java byte code lambda expressions and to combine them while (itr.hasNext()) { int value = ((Row)itr.next()).getInt(0); int mapValue = value + 3; outRow.write(0, mapValue); append(outRow); } // ds: Dataset[Int] … ds.map(value => value + 1) .map(value => value + 2) Dataset will become faster Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki26 Catalyst
Outline • How DataFrame and Dataset are executed? • Deep dive for two cases – Why Dataset is slow – How Spark 2.2 improve performance • Deep dive for another case – Why Dataset is slow • How Dataset are used in ML pipeline? • Next Steps Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki27
How Dataset is Used in ML Pipeline • ML Pipeline is constructed by using DataFrame/Dataset • Algorithm still operates on data in RDD – Conversion overhead between Dataset and RDD val trainingDS: Dataset = ... val tokenizer = new Tokenizer() val hashingTF = new HashingTF() val lr = new LogisticRegression() val pipeline = new Pipeline().setStages( Array(tokenizer, hashingTF, lr)) val model = pipeline.fit(trainingDS) def fit(ds: Dataset[_]) = { ... train(ds) } def train(ds:Dataset[_]):LogisticRegressionModel = { val instances: RDD[Instance] = ds.select(...).rdd.map { // convert DS to RDD case Row(...) => ... } instances.persist(...) instances.treeAggregate(...) } ML pipeline Machine learning algorithm (LogisticRegression) Derived from https://databricks.com/blog/2015/01/07/ml-pipelines-a-new-high-level-api-for-mllib.html Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki28
If We Use Dataset for Machine Learning Algorithm • ML Pipeline is constructed by using DataFrame/Dataset • Algorithm also operates on data in DataFrame/Dataset – No conversion between Dataset and RDD – Optimizations applied by Catalyst – Writing of control flows val trainingDataset: Dataset = ... val tokenizer = new Tokenizer() val hashingTF = new HashingTF() val lr = new LogisticRegression() val pipeline = new Pipeline().setStages( Array(tokenizer, hashingTF, lr)) val model = pipeline.fit(trainingDataset) def train(ds:Dataset[_]):LogisticRegressionModel = { val instances: Dataset[Instance] = ds.select(...).rdd.map { case Row(...) => ... } instances.persist(...) instances.treeAggregate(...) } ML pipeline Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki29 Machine learning algorithm (LogisticRegression)
Next Steps • Achieve further performance improvements for Dataset – Implement Java bytecode analysis and modification • Exploit optimizations in Catalyst (SPARK-14083) • Reduce overhead of data conversion (Our paper) • Exploit SIMD/GPU in Spark by using simple generated code – http://www.spark.tc/simd-and-gpu/ Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki30
Takeaway • Dataset on Spark 2.2 is much faster than on Spark 2.0/2.1 • How Dataset was executed and was improved – Data conversion – Serialization/deserialization – Java bytecode of lambda expression • Continue to improve performance of Dataset • Let us start using Dataset from Today! Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki31
Thank you @kiszk on twitter @kiszk on github http://ibm.biz/ishizaki • @sameeragarwal, @hvanhovell, @gatorsmile, @cloud-fan, @ueshin, @davies, @rxin, @joshrosen, @maropu, @viirya • Spark Technology Center

More Related Content

PDF
Easy, scalable, fault tolerant stream processing with structured streaming - ...
byDatabricks
 
PDF
Spark Summit EU 2016 Keynote - Simplifying Big Data in Apache Spark 2.0
byDatabricks
 
PDF
Productionizing your Streaming Jobs
byDatabricks
 
PDF
SparkSQL: A Compiler from Queries to RDDs
byDatabricks
 
PPTX
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
byDatabricks
 
PDF
Deep Dive Into Catalyst: Apache Spark 2.0’s Optimizer
byDatabricks
 
PDF
Challenging Web-Scale Graph Analytics with Apache Spark with Xiangrui Meng
byDatabricks
 
PDF
Structured Streaming for Columnar Data Warehouses with Jack Gudenkauf
byDatabricks
 
Easy, scalable, fault tolerant stream processing with structured streaming - ...
byDatabricks
 
Spark Summit EU 2016 Keynote - Simplifying Big Data in Apache Spark 2.0
byDatabricks
 
Productionizing your Streaming Jobs
byDatabricks
 
SparkSQL: A Compiler from Queries to RDDs
byDatabricks
 
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
byDatabricks
 
Deep Dive Into Catalyst: Apache Spark 2.0’s Optimizer
byDatabricks
 
Challenging Web-Scale Graph Analytics with Apache Spark with Xiangrui Meng
byDatabricks
 
Structured Streaming for Columnar Data Warehouses with Jack Gudenkauf
byDatabricks
 

What's hot

PDF
Web-Scale Graph Analytics with Apache® Spark™
byDatabricks
 
PDF
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
byDatabricks
 
PDF
Spark Summit EU 2015: Spark DataFrames: Simple and Fast Analysis of Structure...
byDatabricks
 
PDF
Making Structured Streaming Ready for Production
byDatabricks
 
PDF
Data Source API in Spark
byDatabricks
 
PDF
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...
bySpark Summit
 
PDF
Apache Spark Core—Deep Dive—Proper Optimization
byDatabricks
 
PDF
Lessons from the Field, Episode II: Applying Best Practices to Your Apache S...
byDatabricks
 
PDF
Beyond SQL: Speeding up Spark with DataFrames
byDatabricks
 
PDF
Apache® Spark™ 1.6 presented by Databricks co-founder Patrick Wendell
byDatabricks
 
PDF
Jump Start into Apache® Spark™ and Databricks
byDatabricks
 
PDF
Apache Carbondata: An Indexed Columnar File Format for Interactive Query with...
bySpark Summit
 
PPTX
Spark etl
byImran Rashid
 
PDF
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
bySpark Summit
 
PDF
Understanding Query Plans and Spark UIs
byDatabricks
 
PDF
Tuning and Debugging in Apache Spark
byDatabricks
 
PPTX
Building a modern Application with DataFrames
bySpark Summit
 
PPTX
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
bySpark Summit
 
PDF
Enabling exploratory data science with Spark and R
byDatabricks
 
PDF
Jump Start on Apache® Spark™ 2.x with Databricks
byDatabricks
 
Web-Scale Graph Analytics with Apache® Spark™
byDatabricks
 
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
byDatabricks
 
Spark Summit EU 2015: Spark DataFrames: Simple and Fast Analysis of Structure...
byDatabricks
 
Making Structured Streaming Ready for Production
byDatabricks
 
Data Source API in Spark
byDatabricks
 
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...
bySpark Summit
 
Apache Spark Core—Deep Dive—Proper Optimization
byDatabricks
 
Lessons from the Field, Episode II: Applying Best Practices to Your Apache S...
byDatabricks
 
Beyond SQL: Speeding up Spark with DataFrames
byDatabricks
 
Apache® Spark™ 1.6 presented by Databricks co-founder Patrick Wendell
byDatabricks
 
Jump Start into Apache® Spark™ and Databricks
byDatabricks
 
Apache Carbondata: An Indexed Columnar File Format for Interactive Query with...
bySpark Summit
 
Spark etl
byImran Rashid
 
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
bySpark Summit
 
Understanding Query Plans and Spark UIs
byDatabricks
 
Tuning and Debugging in Apache Spark
byDatabricks
 
Building a modern Application with DataFrames
bySpark Summit
 
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
bySpark Summit
 
Enabling exploratory data science with Spark and R
byDatabricks
 
Jump Start on Apache® Spark™ 2.x with Databricks
byDatabricks
 

Similar to Demystifying DataFrame and Dataset with Kazuaki Ishizaki

PDF
Looking back at Spark 2.x and forward to 3.0
byKazuaki Ishizaki
 
PDF
Data Summer Conf 2018, “Hands-on with Apache Spark for Beginners (ENG)” — Akm...
byProvectus
 
PDF
An Introduction to Spark with Scala
byChetan Khatri
 
PDF
icpe2019_ishizaki_public
byKazuaki Ishizaki
 
PDF
HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...
byChetan Khatri
 
PPTX
Building a modern Application with DataFrames
byDatabricks
 
PPTX
DataStax | Data Science with DataStax Enterprise (Brian Hess) | Cassandra Sum...
byDataStax
 
PDF
Introduction to dataset
bydatamantra
 
PDF
Large Scale Data Analytics with Spark and Cassandra on the DSE Platform
byDataStax Academy
 
PDF
Beyond shuffling - Scala Days Berlin 2016
byHolden Karau
 
PDF
Structuring Spark: DataFrames, Datasets, and Streaming
byDatabricks
 
PDF
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...
byDatabricks
 
PDF
Spark Structured APIs
byKnoldus Inc.
 
PDF
Keeping the fun in functional w/ Apache Spark @ Scala Days NYC
byHolden Karau
 
PPTX
Frustration-Reduced PySpark: Data engineering with DataFrames
byIlya Ganelin
 
PDF
Introduction to Spark SQL & Catalyst
byTakuya UESHIN
 
PDF
Introduction to Spark SQL and Catalyst / Spark SQLおよびCalalystの紹介
byscalaconfjp
 
PDF
Jumpstart on Apache Spark 2.2 on Databricks
byDatabricks
 
PPTX
Frustration-Reduced Spark: DataFrames and the Spark Time-Series Library
byIlya Ganelin
 
PDF
Introduction to Spark Datasets - Functional and relational together at last
byHolden Karau
 
Looking back at Spark 2.x and forward to 3.0
byKazuaki Ishizaki
 
Data Summer Conf 2018, “Hands-on with Apache Spark for Beginners (ENG)” — Akm...
byProvectus
 
An Introduction to Spark with Scala
byChetan Khatri
 
icpe2019_ishizaki_public
byKazuaki Ishizaki
 
HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...
byChetan Khatri
 
Building a modern Application with DataFrames
byDatabricks
 
DataStax | Data Science with DataStax Enterprise (Brian Hess) | Cassandra Sum...
byDataStax
 
Introduction to dataset
bydatamantra
 
Large Scale Data Analytics with Spark and Cassandra on the DSE Platform
byDataStax Academy
 
Beyond shuffling - Scala Days Berlin 2016
byHolden Karau
 
Structuring Spark: DataFrames, Datasets, and Streaming
byDatabricks
 
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...
byDatabricks
 
Spark Structured APIs
byKnoldus Inc.
 
Keeping the fun in functional w/ Apache Spark @ Scala Days NYC
byHolden Karau
 
Frustration-Reduced PySpark: Data engineering with DataFrames
byIlya Ganelin
 
Introduction to Spark SQL & Catalyst
byTakuya UESHIN
 
Introduction to Spark SQL and Catalyst / Spark SQLおよびCalalystの紹介
byscalaconfjp
 
Jumpstart on Apache Spark 2.2 on Databricks
byDatabricks
 
Frustration-Reduced Spark: DataFrames and the Spark Time-Series Library
byIlya Ganelin
 
Introduction to Spark Datasets - Functional and relational together at last
byHolden Karau
 

More from Databricks

PPTX
DW Migration Webinar-March 2022.pptx
byDatabricks
 
PPTX
Data Lakehouse Symposium | Day 1 | Part 1
byDatabricks
 
PPT
Data Lakehouse Symposium | Day 1 | Part 2
byDatabricks
 
PPTX
Data Lakehouse Symposium | Day 2
byDatabricks
 
PPTX
Data Lakehouse Symposium | Day 4
byDatabricks
 
PDF
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
byDatabricks
 
PDF
Democratizing Data Quality Through a Centralized Platform
byDatabricks
 
PDF
Learn to Use Databricks for Data Science
byDatabricks
 
PDF
Why APM Is Not the Same As ML Monitoring
byDatabricks
 
PDF
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
byDatabricks
 
PDF
Stage Level Scheduling Improving Big Data and AI Integration
byDatabricks
 
PDF
Simplify Data Conversion from Spark to TensorFlow and PyTorch
byDatabricks
 
PDF
Scaling your Data Pipelines with Apache Spark on Kubernetes
byDatabricks
 
PDF
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
byDatabricks
 
PDF
Sawtooth Windows for Feature Aggregations
byDatabricks
 
PDF
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
byDatabricks
 
PDF
Re-imagine Data Monitoring with whylogs and Spark
byDatabricks
 
PDF
Raven: End-to-end Optimization of ML Prediction Queries
byDatabricks
 
PDF
Processing Large Datasets for ADAS Applications using Apache Spark
byDatabricks
 
PDF
Massive Data Processing in Adobe Using Delta Lake
byDatabricks
 
DW Migration Webinar-March 2022.pptx
byDatabricks
 
Data Lakehouse Symposium | Day 1 | Part 1
byDatabricks
 
Data Lakehouse Symposium | Day 1 | Part 2
byDatabricks
 
Data Lakehouse Symposium | Day 2
byDatabricks
 
Data Lakehouse Symposium | Day 4
byDatabricks
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
byDatabricks
 
Democratizing Data Quality Through a Centralized Platform
byDatabricks
 
Learn to Use Databricks for Data Science
byDatabricks
 
Why APM Is Not the Same As ML Monitoring
byDatabricks
 
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
byDatabricks
 
Stage Level Scheduling Improving Big Data and AI Integration
byDatabricks
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
byDatabricks
 
Scaling your Data Pipelines with Apache Spark on Kubernetes
byDatabricks
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
byDatabricks
 
Sawtooth Windows for Feature Aggregations
byDatabricks
 
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
byDatabricks
 
Re-imagine Data Monitoring with whylogs and Spark
byDatabricks
 
Raven: End-to-end Optimization of ML Prediction Queries
byDatabricks
 
Processing Large Datasets for ADAS Applications using Apache Spark
byDatabricks
 
Massive Data Processing in Adobe Using Delta Lake
byDatabricks
 

Recently uploaded

PPTX
[DSC Europe 25] Hans Kleinsman - The Compliance Gearbox: How Tax Tech Mediate...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Kaja Kandare - LLM as a judge.pptx
byDataScienceConferenc1
 
PDF
[DSC Europe 25] Branko Dzakula - From Defense to Attack: How AI Redefines Cyb...
byDataScienceConferenc1
 
PPTX
IT231_Presenation-Template oftentimes-1.pptx
bygoneweldon026
 
PPTX
[DSC Europe 25] Marko Krstic - Understanding the AI Threat Landscape - Risks,...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Behzad Hosseini - AI Agents in the Wild: Deploying Models tha...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Miodrag Pesovic & Vladislav Radonjic - Federated Data Archite...
byDataScienceConferenc1
 
PDF
Using Data & Technology To Drive Revenue - Fitness KPI 12.25 Webinar
byBryan K. O'Rourke
 
PPTX
[DSC Europe 25] Vladimir Jelic - The AI-Driven Security Shift From Reactive D...
byDataScienceConferenc1
 
PPTX
Talk on Artificial Intelligence_AI Usap Tayo.pptx
byPatrick655831
 
PPTX
[DSC Europe 25] Ivan Peric - Intelligence Swarm Logic and Techno-Functional M...
byDataScienceConferenc1
 
PPTX
Security information presentation and its importance
bybilalkingqwq
 
PPTX
TCP IP and Routing Protocols, Information systems
byryanjamescampos789
 
PPTX
7 Cardiovascular emergencies (Chapter 22) (1).pptx
bysultanlove20044
 
PDF
McCrindle 2026 Trends of the Year
byMark McCrindle
 
PDF
ICT Prep2 Lesson1-2 SM_251002_074210.pdf
bymhalexeg82
 
PDF
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
PPTX
[DSC Europe 25] Branko Urosevic -Rethinking Financial Talent: Integrating Cod...
byDataScienceConferenc1
 
PPTX
Presentation on Hypothesis testing In statistics
bysilversteps03
 
PPTX
cloud computing ppt useful for studestns
byNagarajuJajam
 
[DSC Europe 25] Hans Kleinsman - The Compliance Gearbox: How Tax Tech Mediate...
byDataScienceConferenc1
 
[DSC Europe 25] Kaja Kandare - LLM as a judge.pptx
byDataScienceConferenc1
 
[DSC Europe 25] Branko Dzakula - From Defense to Attack: How AI Redefines Cyb...
byDataScienceConferenc1
 
IT231_Presenation-Template oftentimes-1.pptx
bygoneweldon026
 
[DSC Europe 25] Marko Krstic - Understanding the AI Threat Landscape - Risks,...
byDataScienceConferenc1
 
[DSC Europe 25] Behzad Hosseini - AI Agents in the Wild: Deploying Models tha...
byDataScienceConferenc1
 
[DSC Europe 25] Miodrag Pesovic & Vladislav Radonjic - Federated Data Archite...
byDataScienceConferenc1
 
Using Data & Technology To Drive Revenue - Fitness KPI 12.25 Webinar
byBryan K. O'Rourke
 
[DSC Europe 25] Vladimir Jelic - The AI-Driven Security Shift From Reactive D...
byDataScienceConferenc1
 
Talk on Artificial Intelligence_AI Usap Tayo.pptx
byPatrick655831
 
[DSC Europe 25] Ivan Peric - Intelligence Swarm Logic and Techno-Functional M...
byDataScienceConferenc1
 
Security information presentation and its importance
bybilalkingqwq
 
TCP IP and Routing Protocols, Information systems
byryanjamescampos789
 
7 Cardiovascular emergencies (Chapter 22) (1).pptx
bysultanlove20044
 
McCrindle 2026 Trends of the Year
byMark McCrindle
 
ICT Prep2 Lesson1-2 SM_251002_074210.pdf
bymhalexeg82
 
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
[DSC Europe 25] Branko Urosevic -Rethinking Financial Talent: Integrating Cod...
byDataScienceConferenc1
 
Presentation on Hypothesis testing In statistics
bysilversteps03
 
cloud computing ppt useful for studestns
byNagarajuJajam
 

Demystifying DataFrame and Dataset with Kazuaki Ishizaki

  • 1.
    Kazuaki Ishizaki IBM Research– Tokyo @kiszk Demystifying DataFrame and Dataset #SFdev20
  • 2.
    About Me –Kazuaki Ishizaki • Researcher at IBM Research in compiler optimizations • Working for IBM Java virtual machine over 20 years – In particular, just-in-time compiler • Contributor to SQL package in Apache Spark • Committer of GPUEnabler – Apache Spark Plug-in to execute GPU code on Spark • https://github.com/IBMSparkGPU/GPUEnabler • Homepage: http://ibm.biz/ishizaki • Github: https://github.com/kiszk, Twitter: @kiszk Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki2
  • 3.
    Spark 2.2 makesDataset Faster Compared to Spark 2.0 (also 2.1) • 1.6x faster for map() with scalar variable • 4.5x faster for map() with primitive array Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki3 ds = (1 to ...).toDS.cache ds.map(value => value + 1) ds = Seq(Array(…), Array(…), …).toDS.cache ds.map(array => Array(array(0))) from enhancements in Catalyst optimizer and codegen
  • 4.
    How Does ItMatter? • Application programmers – ML pipeline will become faster • Library developers – You will want to use Dataset instead of RDD • SQL package developers – You will know detail on Dataset Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki4
  • 5.
    DataFrame, Dataset, andRDD df = (1 to 100).toDF.cache df.selectExpr(“value + 1”) ds = (1 to 100).toDS.cache ds.map(value => value + 1) SQLDataFrame (DF) Dataset (DS) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki5 RDD rdd = sc.parallelize( (1 to 100)).cache rdd.map(value => value + 1) Based on lambda expressionBased on SQL operation
  • 6.
    How DF, DS,and RDD Work • We expect the same performance on DF and DS From Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Michael Armbrust DS DF Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki6 Catalyst // generated Java code // for DF and DS while (itr.hasNext()) { Row r = (Row)itr.next(); int v = r.getInt(0); … }
  • 7.
    Dataset for IntValue is Slow on Spark 2.0 • 1.7x slow for addition to scalar int value 0 0.5 1 1.5 2 Relative execution time over DataFrame DataFrame Dataset df.selectExpr(“value + 1”) ds.map(value => value + 1) All experiments are done using 16-core Intel E2667-v3 3.2GHz, with 384GB mem, Ubuntu 16.04, OpenJDK 1.8.0u212-b13 with 256GB heap, Spark master=local[1] Spark branch-2.0 and branch-2.2, ds/df/rdd are cached Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki7 Shorter is better
  • 8.
    Dataset for IntArray is Extremely Slow on Spark 2.0 • 54x slow for creating a new array for primitive int array 0 10 20 30 40 50 60 Relative execution time over DataFrame DataFrame Dataset df = Seq(Array(…), Array(…), …) .toDF(“a”).cache df.selectExpr(“Array(a[0])”) ds = Seq(Array(…), Array(…), …) .toDS.cache ds.map(a => Array(a(0))) SPARK-16070 pointed out this performance problem Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki8 Shorter is better
  • 9.
    Outline • How DataFrameand Dataset are executed? • Deep dive for two cases – Why Dataset was slow – How Spark 2.2 improves performance • Deep dive for another case – Why Dataset is slow – How future Spark will improve performance • How Dataset is used in ML pipeline? • Next Steps Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki9
  • 10.
    Simple Generated Codefor int with DF • Generated code performs `+` to int – Use strongly-typed ‘int’ variable 1: while (itr.hasNext()) { // execute a row 2: // get a value from a row in DF 3: int value =((Row)itr.next()).getInt(0); 4: // compute a new value 5: int mapValue = value + 1; 6: // store a new value to a row in DF 7: outRow.write(0, mapValue); 8: append(outRow); 9: } // df: DataFrame for int … df.selectExpr(“value + 1”) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki10 Catalyst generates Java code Catalyst
  • 11.
    Weird Generated Codewith DS • Generated code performs four data conversions – Use standard API with generic type method apply(Object) while (itr.hasNext()) { int value = ((Row)itr.next()).getInt(0); Object objValue = new Integer(value); int mapValue = ((Integer) mapFunc.apply(objVal)).toValue(); outRow.write(0, mapValue); append(outRow); } Object apply(Object obj) { int value = Integer(obj).toValue(); return new Integer(apply$II(value)); } int apply$II(int value) { return value + 1; } // ds: Dataset[Int] … ds.map(value => value + 1) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki11 Catalyst generates Java code Scalac generates these functions from lambda expression Catalyst
  • 12.
    Simple Generated Codewith DS on Spark 2.2 • SPARK-19008 enables generated code to use int value – Use non-standard API with type-specialized method apply$II(int) while (itr.hasNext()) { int value = ((Row)itr.next()).getInt(0); int mapValue = mapFunc.apply$II(value); outRow.write(0, mapValue); append(outRow); } int apply$II(int value) { return value + 1;} // ds: Dataset[Int] … ds.map(value => value + 1) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki12 Catalyst Scalac generates this function from lambda expression
  • 13.
    Dataset is NotSlow on Spark 2.2 0 0.5 1 1.5 2 Relative execution time over DataFrame DataFrame Dataset df.selectExpr(“value + 1”) ds.map(value => value + 1) Shorter is better Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki13 1.6x • Improve performance by 1.6x compared to Spark 2.0 – DataFrame is 9% faster than Dataset
  • 14.
    RDD/DF/DS achieve close performanceon Spark 2.2 • RDD is 5% faster than DataFrame 0 0.5 1 1.5 2 Relative execution time over DataFrame RDD DataFrame Dataset df.selectExpr(“value + 1”) ds.map(value => value + 1) rdd.map(value => value + 1) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki14 9% 5% Shorter is better
  • 15.
    Simple Generated Codefor Array with DF • All operations access data in internal data format for array ArrayData (devised in Project Tungsten) // df: DataFrame for Array(Int) … df.selectExpr(“Array(a[0])”) ArrayData inArray, outArray; while (itr.hasNext()) { inArray = ((Row)itr.next()).getArray(0); int value = inArray.getInt(0)); //a[0] outArray = new UnsafeArrayData(); ... outArray.setInt(0, value); //Array(a[0]) outArray.writeToMemory(outRow); append(outRow); } Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki15 Catalyst generates Java code Internal data format (ArrayData) inArray.getInt(0) outArray.setInt(0, …) Catalyst
  • 16.
    Lambda Expression withDS requires Java Object • Need serialization/deserialzation (Ser/De) between internal data format and Java object for lambda expression // ds: DataSet[Array(Int)] … ds.map(a => Array(a(0))) ArrayData inArray, outArray; while (itr.hasNext()) { inArray = ((Row)itr.next()).getArray(0); append(outRow); } Internal data formatJava object Ser/De Array(a(0)) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki16 int[] mapArray = Array(a(0)); Java bytecode for lambda expression Catalyst Deserialization Serialization Ser/De
  • 17.
    ArrayData inArray, outArray; while(itr.hasNext()) { inArray = ((Row)itr.next().getArray(0); Object[] tmp = new Object[inArray.numElements()]; for (int i = 0; i < tmp.length; i ++) { tmp[i] = (inArray.isNullAt(i)) ? Null : inArray.getInt(i); } ArrayData array = new GenericIntArrayData(tmpArray); int[] javaArray = array.toIntArray(); int[] mapArray = (int[])map_func.apply(javaArray); outArray = new GenericArrayData(mapArray); for (int i = 0; i < outArray.numElements(); i++) { if (outArray.isNullAt(i)) { arrayWriter.setNullInt(i); } else { arrayWriter.write(i, outArray.getInt(i)); } } append(outRow); } ArrayData inArray, outArray; while (itr.hasNext()) { inArray = ((Row)itr.next().getArray(0); append(outRow); } Extremely Weird Code for Array with DS on Spark 2.0 • Data conversion is too slow • Element-wise copy is slow // ds: DataSet[Array(Int)] … ds.map(a => Array(a(0))) Data conversion Element-wise data copy Element-wise data copy Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki17 Data conversion Copy with Java object creation Element-wise data copy Copy each element with null check int[] mapArray = Array(a(0)); Catalyst Data conversion Element-wise data copy Ser De
  • 18.
    Simple Generated Codefor Array on Spark 2.2 • SPARK-15985, SPARK-17490, and SPARK-15962 simplify Ser/De by using bulk data copy – Data conversion and element-wise copy are not used – Bulk copy is faster // ds: DataSet[Array(Int)] … ds.map(a => Array(a(0))) while (itr.hasNext()) { inArray =((Row)itr.next()).getArray(0); int[] javaArray = inArray.toIntArray(); int[] mapArray = (int[])mapFunc.apply(javaArray); outArray = UnsafeArrayData .fromPrimitiveArray(mapArray); outArray.writeToMemory(outRow); append(outRow); } Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki18 Bulk data copy Copy whole array using memcpy() Catalyst while (itr.hasNext()) { inArray =((Row)itr.next()).getArray(0); int[]mapArray=(int[])map.apply(javaArray); append(outRow); } Bulk data copy int[] mapArray = Array(a(0)); Bulk data copy Bulk data copy
  • 19.
    0 10 2030 40 50 60 Relative execution time over DataFrame DataFrame Dataset Dataset for Array is Not Extremely Slow over DataFrame • Improve performance by 4.5 compared to Spark 2.0 – DataFrame is 12x faster than Dataset ds = Seq(Array(…), Array(…), …) .toDS.cache ds.map(a => Array(a(0))) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki19 4.5x df = Seq(Array(…), Array(…), …) .toDF（”a”).cache df.selectExpr(“Array(a[0])”) Would this overhead be negligible if map() is much computation intensive? 12x Shorter is better
  • 20.
    RDD and DataFramefor Array Achieve Close Performance • DataFrame is 1.2x faster than RDD 0 5 10 15 Relative execution time over DataFrame RDD DataFrame Dataset ds = Seq(Array(…), Array(…), …) .toDS.cache ds.map(a => Array(a(0))) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki20 df = Seq(Array(…), Array(…), …) .toDF（”a”).cache df.selectExpr(“Array(a[0])”) rdd = sc.parallelize(Seq( Array(…), Array(…), …)).cache rdd.map(a => Array(a(0))) 1.2 12 Shorter is better
  • 21.
    Enable Lambda Expressionto Use Internal Data Format • Our prototype modifies Java bytecode of lambda expression to access internal data format – We improved performance by avoiding Ser/De // ds: DataSet[Array(Int)] … ds.map(a => Array(a(0))) while (itr.hasNext()) { inArray = ((Row)itr.next().getArray(0); outArray = mapFunc.applyTungsten(inArray); outArray.writeToMemory(outRow); append(outRow); } Internal data format “Accelerating Spark Datasets by inlining deserialization”, our academic paper at IPDPS2017 1 1 apply(Object) 1011 applyTungsten(ArrayData) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki21 Catalyst generates Java code Catalyst Our prototype
  • 22.
    Outline • How DataFrameand Dataset are executed? • Deep dive for two cases – Why Dataset is slow – How Spark 2.2 improve performance • Deep dive for another case – Why Dataset is slow • How Dataset are used in ML pipeline? • Next Steps Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki22
  • 23.
    0 0.5 11.5 Relative execution time over DataFrame RDD DataFrame Dataset Dataset for Two Scalar Adds is Slow on Spark 2.2 • DataFrame and Dataset are faster than RDD • Dataset is 1.1x slower than DataFrame df.selectExpr(“value + 1”) .selectExpr(“value + 2”) ds.map(value => value + 1) .map(value => value + 2) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki23 rdd.map(value => value + 1) .map(value => value + 2) Shorter is better ☺  1.1 1.4
  • 24.
    Adds are Combinedwith DF • Spark 2.2 understands operations in selectExpr() and combines two adds into one add ‘value + 3’ while (itr.hasNext()) { Row row = (Row)itr.next(); int value = row.getInt(0); int mapValue = value + 3; projectRow.write(0, mapValue); append(projectRow); } // df: DataFrame for Int … df.selectExpr(“value + 1”) .selectExpr(“value + 2”) Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki24 Catalyst Catalyst generates Java code
  • 25.
    Two Method Callsfor Adds Remain with DS • Spark 2.2 cannot understand lambda expressions stored as Java bytecode while (itr.hasNext()) { int value = ((Row)itr.next()).getInt(0); int mapValue = mapFunc1.apply$II(value); mapValue += mapFunc2.apply$II(mapValue); outRow.write(0, mapValue); append(outRow); } // ds: Dataset[Int] … ds.map(value => value + 1) .map(value => value + 2) class MapFunc1 { int apply$II(int value) { return value + 1;}} class MapFunc2 { int apply$II(int value) { return value + 2;}} Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki25 Catalyst Scalac generates these functions from lambda expression
  • 26.
    class MapFunc1 { intapply$II(int value) { return value + 1;}} class MapFunc2 { int apply$II(int value) { return value + 2;}} Adds Will be Combined on Future Spark 2.x • SPARK-14083 will allow future Spark to understand Java byte code lambda expressions and to combine them while (itr.hasNext()) { int value = ((Row)itr.next()).getInt(0); int mapValue = value + 3; outRow.write(0, mapValue); append(outRow); } // ds: Dataset[Int] … ds.map(value => value + 1) .map(value => value + 2) Dataset will become faster Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki26 Catalyst
  • 27.
    Outline • How DataFrameand Dataset are executed? • Deep dive for two cases – Why Dataset is slow – How Spark 2.2 improve performance • Deep dive for another case – Why Dataset is slow • How Dataset are used in ML pipeline? • Next Steps Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki27
  • 28.
    How Dataset isUsed in ML Pipeline • ML Pipeline is constructed by using DataFrame/Dataset • Algorithm still operates on data in RDD – Conversion overhead between Dataset and RDD val trainingDS: Dataset = ... val tokenizer = new Tokenizer() val hashingTF = new HashingTF() val lr = new LogisticRegression() val pipeline = new Pipeline().setStages( Array(tokenizer, hashingTF, lr)) val model = pipeline.fit(trainingDS) def fit(ds: Dataset[_]) = { ... train(ds) } def train(ds:Dataset[_]):LogisticRegressionModel = { val instances: RDD[Instance] = ds.select(...).rdd.map { // convert DS to RDD case Row(...) => ... } instances.persist(...) instances.treeAggregate(...) } ML pipeline Machine learning algorithm (LogisticRegression) Derived from https://databricks.com/blog/2015/01/07/ml-pipelines-a-new-high-level-api-for-mllib.html Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki28
  • 29.
    If We UseDataset for Machine Learning Algorithm • ML Pipeline is constructed by using DataFrame/Dataset • Algorithm also operates on data in DataFrame/Dataset – No conversion between Dataset and RDD – Optimizations applied by Catalyst – Writing of control flows val trainingDataset: Dataset = ... val tokenizer = new Tokenizer() val hashingTF = new HashingTF() val lr = new LogisticRegression() val pipeline = new Pipeline().setStages( Array(tokenizer, hashingTF, lr)) val model = pipeline.fit(trainingDataset) def train(ds:Dataset[_]):LogisticRegressionModel = { val instances: Dataset[Instance] = ds.select(...).rdd.map { case Row(...) => ... } instances.persist(...) instances.treeAggregate(...) } ML pipeline Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki29 Machine learning algorithm (LogisticRegression)
  • 30.
    Next Steps • Achievefurther performance improvements for Dataset – Implement Java bytecode analysis and modification • Exploit optimizations in Catalyst (SPARK-14083) • Reduce overhead of data conversion (Our paper) • Exploit SIMD/GPU in Spark by using simple generated code – http://www.spark.tc/simd-and-gpu/ Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki30
  • 31.
    Takeaway • Dataset onSpark 2.2 is much faster than on Spark 2.0/2.1 • How Dataset was executed and was improved – Data conversion – Serialization/deserialization – Java bytecode of lambda expression • Continue to improve performance of Dataset • Let us start using Dataset from Today! Demystifying DataFrame and Dataset (#SFdev20) / Kazuaki Ishizaki31
  • 32.
    Thank you @kiszk ontwitter @kiszk on github http://ibm.biz/ishizaki • @sameeragarwal, @hvanhovell, @gatorsmile, @cloud-fan, @ueshin, @davies, @rxin, @joshrosen, @maropu, @viirya • Spark Technology Center