SlideShare a Scribd company logo
Spark + Parquet in Depth
Robbie Strickland
VP, Engines & Pipelines, Watson Data Platform
@rs_atl
Emily May Curtin
Software Engineer, IBM Spark Technology Center
@emilymaycurtin
Atlanta…
Atlanta!!!!
Outline
• Why Parquet
• Parquet by example
• How Parquet works
• How Spark squeezes out efficiency
• What’s the catch
• Tuning tips for Spark + Parquet
Goals for Data Lake Storage
• Good Usability
– Easy to backup
– Minimal learning curve
– Easy integration with
existing tools
• Resource efficient
– Disk space
– Disk I/O Time
– Network I/O
• AFFORDABLE
• CA$$$H MONEY
• DEVELOPER HOURS à $$$
• COMPUTE CYCLES à $$$
•FAST QUERIES
Little Costs Matter at Actual Scale
“Very Large Dataset” Weather-Scale Data
Disk and Network I/O Hurt
Action Computer Time “Human Scale” Time
1 CPU cycle 0.3 ns 1 s
Level 1 cache access 0.9 ns 3 s
Level 2 cache access 2.8 ns 9 s
Level 3 cache access 12.9 ns 43 s
Main memory access 120 ns 6 min
Solid-state disk I/O 50-150 μs 2-6 days
Rotational disk I/O 1-10 ms 1-12 months
Internet: SF to NYC 40 ms 4 years
Internet: SF to UK 81 ms 8 years
Internet: SF to Australia 183 ms 19 years
Source: Systems Performance: Enterprise and the Cloud by Brendan Gregg via CodingHorror.com “The Infinite Space Between Words”
Options For Multi-PB Data Lake Storage
Files Compressed Files Databases
Usability Great! Great! OK to BAD (not as easy as a file!)
Administration None! None! LOTS
Spark Integration Great! Great! Varies
Resource Efficiency BAD (Big storage, heavy I/O) OK… (Less storage) BAD (Requires storage AND CPU)
Scalability Good-ish Good-ish BAD (For multi-petabyte!)
CO$$$$T OK… OK… TERRIBLE
QUERY TIME TERRIBLE BAD Good!
CSV vs. Parquet Column Selection Query
2892.3
50.6 43.4 40.3 28.90
500
1000
1500
2000
2500
3000
CSV Parquet: LZO Parquet:
Uncompressed
Parquet: GZIP Parquet: Snappy
Query Time (seconds)
SELECT cacheStatus, bytesSent from ADatasetThatHasToDoWithCDNs
WHERE cacheStatus LIKE 'stale'
AND bytesSent < 500
CSV vs. Parquet Table Scan Query
2059
50 49.1 44.2 39.60
500
1000
1500
2000
CSV Parquet: LZO Parquet: Snappy Parquet:
Uncompressed
Parquet: GZIP
Query Time (seconds)
SELECT * from ADatasetThatHasToDoWithCDNs
WHERE cacheStatus LIKE 'stale'
AND bytesSent < 500
Parquet Format
“Apache Parquet is a columnar storage format available to any project in the Hadoop
ecosystem, regardless of the choice of data processing framework, data model or
programming language.”
• Binary Format
• API for JVM/Hadoop & C++
• Columnar
• Encoded
• Compressed
• Machine-Friendly
Parquet By Example
Introducing the Dataset
Very Important Dataset
Title Released Label PeakChart.UK Certification.BVMI Certification.RIAA (omitted for space…)
Led Zeppelin 01/12/1969 Atlantic 6 8x Platinum …
Led Zeppelin II 10/22/1969 Atlantic 1 Platinum Diamond …
Led Zeppelin III 10/05/1970 Atlantic 1 Gold 6x Platinum …
Led Zeppelin IV 11/08/1971 Atlantic 1 3x Gold Diamond …
Houses of the Holy 03/28/1973 Atlantic 1 Gold Diamond …
Physical Graffiti 02/24/1975 Swan Song 1 Gold Diamond …
Presence 03/31/1976 Swan Song 1 3x Platinum …
In Through The
Out Door
08/15/1979 Swan Song 1 6x Platinum …
Coda 11/19/1982 Swan Song 4 Platinum …
One Row, Two Different Ways
{
"Title" : "Led Zeppelin IV",
"Released" : "11/8/1971",
"Label" : "Atlantic",
"PeakChart.UK" : 1,
"PeakChart.AUS" : 2,
"PeakChart.US" : 2,
"Certification.ARIA" : "9x Platinum",
"Certification.BPI" : "6x Platinum",
"Certification.BVMI" : "3x Gold",
"Certification.CRIA" : "2x Diamond",
"Certification.IFPI" : "2x Platinum",
"Certification.NVPI" : "Platinum",
"Certification.RIAA" : "Diamond",
"Certification.SNEP" : "2x Platinum"
}
{
"TITLE": "LED ZEPPELIN IV",
"RELEASED": "11/8/1971",
"LABEL": "ATLANTIC",
"PEAKCHART": {
"UK": 1,
"AUS": 2,
"US": 2 },
"CERTIFICATION": {
"ARIA": "9X PLATINUM",
"BPI": "6X PLATINUM",
"BVMI": "3X GOLD",
"CRIA": "2X DIAMOND",
"IFPI": "2X PLATINUM",
"NVPI": "PLATINUM",
"RIAA": "DIAMOND",
"SNEP": "2X PLATINUM“ }
}
The Flat Schema Data
Title Released Label PeakChart.UK PeakChart.AUS PeakChart.US PeakChart.Mars
Certification.ARIA Certification.BPI Certification.BVMI Certification.CRIA Certification.IFPI
Certification.NVPI Certification.RIAA Certification.SNEP
Led Zeppelin 01/12/1969 Atlantic 6 9 10 2x Platinum 2x Platinum Diamond Gold
Gold 8x Platinum Gold
Led Zeppelin II 10/22/1969 Atlantic 1 1 1 4x Platinum 4x Platinum Platinum 9x
Platinum Gold Diamond Platinum
Led Zeppelin III 10/5/1970 Atlantic 1 1 1 Platinum Gold 3x Platinum Gold Gold
6x Platinum Platinum
Led Zeppelin IV 11/8/1971 Atlantic 1 2 2 9x Platinum 6x Platinum 3x Gold 2x
Diamond 2x Platinum Platinum Diamond 2x Platinum
Houses of the Holy 03/28/1973 Atlantic 1 1 1 Platinum Gold Diamond Gold
Physical Graffiti 02/24/1975 Swan Song 1 1 1 3x Platinum 2x Platinum Gold
Diamond Gold
Presence 03/31/1976 Swan Song 1 4 1 Platinum 3x Platinum
In Through The Out Door 08/15/1979 Swan Song 1 3 1 2x Platinum Platinum 6x
Platinum
Coda 11/19/1982 Swan Song 4 9 6 Silver Platinum
The Nested Schema Data
{"Title":"Led Zeppelin", "Released":"01/12/1969", "Label":"Atlantic", "PeakChart":{"UK":6, "AUS":9,
"US":10}, "Certification":{"ARIA":"2x Platinum", "BPI":"2x Platinum", "CRIA":"Diamond", "IFPI":"Gold",
"NVPI":"Gold", "RIAA":"8x Platinum", "SNEP":"Gold"}}
{"Title":"Led Zeppelin II", "Released":"10/22/1969", "Label":"Atlantic", "PeakChart":{"UK":1, "AUS":1,
"US":1}, "Certification":{"ARIA":"4x Platinum", "BPI":"4x Platinum", "BVMI":"Platinum", "CRIA":"9x
Platinum", "IFPI":"Gold", "RIAA":"Diamond", "SNEP":"Platinum"}}
{"Title":"Led Zeppelin III", "Released":"10/5/1970", "Label":"Atlantic", "PeakChart":{"UK":1, "AUS":1,
"US":1}, "Certification":{"BPI":"Platinum", "BVMI":"Gold", "CRIA":"3x Platinum", "IFPI":"Gold",
"NVPI":"Gold", "RIAA":"6x Platinum", "SNEP":"Platinum"}}
{"Title":"Led Zeppelin IV", "Released":"11/8/1971", "Label":"Atlantic", "PeakChart":{"UK":1, "AUS":2,
"US":2}, "Certification":{"ARIA":"9x Platinum", "BPI":"6x Platinum", "BVMI":"3x Gold", "CRIA":"2x
Diamond", "IFPI":"2x Platinum", "NVPI":"Platinum", "RIAA":"Diamond", "SNEP":"2x Platinum"}}
{"Title":"Houses of the Holy", "Released":"03/28/1973", "Label":"Atlantic", "PeakChart":{"UK":1,
"AUS":1, "US":1}, "Certification":{"BPI":"Platinum", "BVMI":"Gold", "RIAA":"Diamond", "SNEP":"Gold"}}
{"Title":"Physical Graffiti", "Released":"02/24/1975", "Label":"Swan Song", "PeakChart":{"UK":1,
"AUS":1, "US":1}, "Certification":{"ARIA":"3x Platinum", "BPI":"2x Platinum", "BVMI":"Gold",
"RIAA":"Diamond", "SNEP":"Gold"}}
{"Title":"Presence", "Released":"03/31/1976", "Label":"Swan Song", "PeakChart":{"UK":1, "AUS":4,
"US":1}, "Certification":{"BPI":"Platinum", "RIAA":"3x Platinum"}}
{"Title":"In Through The Out Door", "Released":"08/15/1979", "Label":"Swan Song", "PeakChart":{"UK":1,
"AUS":3, "US":1}, "Certification":{"ARIA":"2x Platinum", "BPI":"Platinum", "RIAA":"6x Platinum"}}
{"Title":"Coda", "Released":"11/19/1982", "Label":"Swan Song", "PeakChart":{"UK":4, "AUS":9, "US":6},
"Certification":{"BPI":"Silver", "RIAA":"Platinum"}}
Parquet By Example
Writing Parquet Using Spark
Writing To Parquet: Flat Schema
val flatDF = spark
.read.option("delimiter", "t")
.option("header", "true").csv(flatInput)
.rdd
.map(r => transformRow(r))
.toDF
flatDF.write
.option("compression", "snappy")
.parquet(flatOutput)
Writing To Parquet: Flat Schema
/*Oh crap, the Ints are gonna get pulled in as Strings unless we transform*/
case class LedZeppelinFlat(
Title: Option[String],
Released: Option[String],
Label: Option[String],
UK: Option[Int],
AUS: Option[Int],
US: Option[Int],
ARIA: Option[String],
BPI: Option[String],
BVMI: Option[String],
CRIA: Option[String],
IFPI: Option[String],
NVPI: Option[String],
RIAA: Option[String],
SNEP: Option[String]
)
Writing To Parquet: Flat Schema
def transformRow(r: Row): LedZeppelinFlat = {
def getStr(r: Row, i: Int) = if(!r.isNullAt(i)) Some(r.getString(i)) else None
def getInt(r: Row, i: Int) = if(!r.isNullAt(i)) Some(r.getInt(i)) else None
LedZeppelinFlat(
getStr(r, 0),
getStr(r, 1),
getStr(r, 2),
getInt(r, 3),
getInt(r, 4),
getInt(r, 5),
getStr(r, 7),
getStr(r, 8),
getStr(r, 9),
getStr(r, 10),
getStr(r, 11),
getStr(r, 12),
getStr(r, 13),
getStr(r, 14)
)
}
Writing To Parquet: Flat Schema
val outDF = spark
.read.option("delimiter", "t")
.option("header", "true").csv(flatInput)
.rdd
.map(r => transformRow(r))
.toDF
outDF.write
.option("compression", "snappy")
.parquet(flatOutput)
Writing To Parquet: Flat Schema
Writing To Parquet: Flat Schema…
In Java
Writing To Parquet: Flat Schema…
With MapReduce
Writing To Parquet: Nested Schema
val nestedDF = spark.read.json(nestedInput)
nestedDF.write
.option("compression", "snappy")
.parquet(nestedOutput)
Writing To Parquet: Nested Schema
Parquet By Example
Let’s See An Example!
Parquet Schema Two Different Ways
Nested Schema
Title: OPTIONAL BINARY O:UTF8 R:0 D:1
Released: OPTIONAL BINARY O:UTF8 R:0 D:1
Label: OPTIONAL BINARY O:UTF8 R:0 D:1
PeakChart: OPTIONAL F:3
.AUS: OPTIONAL INT64 R:0 D:2
.UK: OPTIONAL INT64 R:0 D:2
.US: OPTIONAL INT64 R:0 D:2
Certification: OPTIONAL F:8
.ARIA: OPTIONAL BINARY O:UTF8 R:0 D:2
.BPI: OPTIONAL BINARY O:UTF8 R:0 D:2
.BVMI: OPTIONAL BINARY O:UTF8 R:0 D:2
.CRIA: OPTIONAL BINARY O:UTF8 R:0 D:2
.IFPI: OPTIONAL BINARY O:UTF8 R:0 D:2
.NVPI: OPTIONAL BINARY O:UTF8 R:0 D:2
.RIAA: OPTIONAL BINARY O:UTF8 R:0 D:2
.SNEP: OPTIONAL BINARY O:UTF8 R:0 D:2
FLAT SCHEMA
TITLE: OPTIONAL BINARY O:UTF8 R:0 D:1
RELEASED: OPTIONAL BINARY O:UTF8 R:0 D:1
LABEL: OPTIONAL BINARY O:UTF8 R:0 D:1
PEAKCHART.UK: REQUIRED INT32 R:0 D:0
PEAKCHART.AUS: REQUIRED INT32 R:0 D:0
PEAKCHART.US: REQUIRED INT32 R:0 D:0
CERTIFICATION.ARIA: OPTIONAL BINARY O:UTF8 R:0 D:1
CERTIFICATION.BPI: OPTIONAL BINARY O:UTF8 R:0 D:1
CERTIFICATION.BVMI: OPTIONAL BINARY O:UTF8 R:0 D:1
CERTIFICATION.CRIA: OPTIONAL BINARY O:UTF8 R:0 D:1
CERTIFICATION.IFPI: OPTIONAL BINARY O:UTF8 R:0 D:1
CERTIFICATION.NVPI: OPTIONAL BINARY O:UTF8 R:0 D:1
CERTIFICATION.RIAA: OPTIONAL BINARY O:UTF8 R:0 D:1
CERTIFICATION.SNEP: OPTIONAL BINARY O:UTF8 R:0 D:1
Schema Breakdown
COLUMN NAME Title
OPTIONAL / REQUIRED / REPEATED OPTIONAL
DATA TYPE BINARY
ENCODING INFO FOR BINARY 0:UTF8
REPETITION VALUE R:0
DEFINITION VALUE D:0
FLAT SCHEMA
TITLE: OPTIONAL BINARY O:UTF8 R:0 D:1
RELEASED: OPTIONAL BINARY O:UTF8 R:0 D:1
LABEL: OPTIONAL BINARY O:UTF8 R:0 D:1
PEAKCHART.UK: REQUIRED INT32 R:0 D:0
. . .
Repetition and Definition Levels
Source: https://github.com/apache/parquet-mr
One Parquet Row, Two Ways
Title = Led Zeppelin IV
Released = 11/8/1971
Label = Atlantic
PeakChart:
.AUS = 2
.UK = 1
.US = 2
Certification:
.ARIA = 9x Platinum
.BPI = 6x Platinum
.BVMI = 3x Gold
.CRIA = 2x Diamond
.IFPI = 2x Platinum
.NVPI = Platinum
.RIAA = Diamond
.SNEP = 2x Platinum
TITLE = LED ZEPPELIN IV
RELEASED = 11/8/1971
LABEL = ATLANTIC
PEAKCHART.UK = 1
PEAKCHART.AUS = 2
PEAKCHART.US = 2
CERTIFICATION.ARIA = 9X PLATINUM
CERTIFICATION.BPI = 6X PLATINUM
CERTIFICATION.BVMI = 3X GOLD
CERTIFICATION.CRIA = 2X DIAMOND
CERTIFICATION.IFPI = 2X PLATINUM
CERTIFICATION.NVPI = PLATINUM
CERTIFICATION.RIAA = DIAMOND
CERTIFICATION.SNEP = 2X PLATINUM
Parquet By Example
Reading and Querying Using Spark
Slightly Different Queries
// Many ways, this is just one!
val flatParquet = "s3a://…../LedZeppelin-FlatSchema.parquet/"
val flatdf = spark.read.parquet(flatParquet)
flatdf.createOrReplaceTempView("LedZeppelinFlat")
val nestedParquet = "s3a://…../LedZeppelin-NestedSchema.parquet/"
val nesteddf = spark.read.parquet(nestedParquet)
nesteddf.createOrReplaceTempView("LedZeppelinNested")
val flatQuery= "select Title, US from LedZeppelinFlat where US = 1"
val nestedQuery = "select Title, PeakChart.US from LedZeppelinNested
where PeakChart.US = 1"
spark.sql(flatQuery)
spark.sql(nestedQuery)
Same Result
How Parquet Works
Parquet Structure In the Filesystem
led-zeppelin-albums.parquet/
• _SUCCESS
• _common_metadata
• _metadata
• Year=1969/
– Part-r-00000-6d4d42e2-c13f-4bdf-917d-2152b24a0f24.snappy.parquet
– Part-r-00001-6d4d42e2-c13f-4bdf-917d-2152b24a0f24.snappy.parquet
– …
• Year=1970/
– Part-r-00000-35cb7ef4-6de6-4efa-9bc6-5286de520af7.snappy.parquet
– ...
• Groups of rows, partitioned by column values, compressed however you like. (GZIP, LZO, Snappy, etc)
• In general LZO wins size benchmarks, Snappy good balance between size and CPU intensity.
Data In Columns On Disk
Led Zeppelin IV 11/08/1971 1 Houses of the Holy 03/28/1973 1 Physical Graffiti 02/24/1975 1
Led Zeppelin IV 11/08/1971 1Houses of the Holy 03/28/1973 1Physical Graffiti 02/24/1975 1
Row-Oriented data on disk
Column-Oriented data on disk
Title Date Chart
Encoding: Incremental Encoding
Led_Zeppelin_IV
Led_Zeppelin_III
Led_Zeppelin_II
Led_Zeppelin 0 Led_Zeppelin
12 _II
15 I
14 V
58 bytes* 24 bytes*
*not counting delimiters
ENCODING
58% Reduction
Encoding: Dictionary Encoding
ENCODING
Atlantic
Swan Song
Atlantic
Atlantic
Atlantic
Atlantic
Atlantic
Swan Song
Swan Song
Swan Song
84 bytes*
0
1
0
0
0
0
0
1
1
1
0 à Atlantic
1 à Swan Song
1.25 bytes + dictionary size
~98% Reduction
More Encoding Schemes
• Plain (bit-packed, little endian, etc)
• Dictionary Encoding
• Run Length Encoding/Bit Packing Hybrid
• Delta Encoding
• Delta-Length Byte Array
• Delta Strings (incremental Encoding)
See https://github.com/apache/parquet-format/blob/master/Encodings.md for more detail
Slicing and Dicing Within A Compressed File
File Metadata
Row
Group Row Group
Column
Chunk
Col1
Column
Chunk
Col2A
Column
Chunk
Col2B
Page
Header
Column Chunk Column 3
Page Header
Page Page Page
- Metadata
- R Values
- D Values
- ENCODED DATA
Slicing and Dicing Within A Compressed File
File Metadata
Row
Group Row Group
Column
Chunk
Col1
Column
Chunk
Col2A
Column
Chunk
Col2B
Page
Header
Column Chunk Column 3
Page Header
Page Page Page
- Metadata
- R Values
- D Values
- ENCODED DATA
Slicing and Dicing Within A Compressed File
File Metadata
Row
Group Row Group
Column
Chunk
Col1
Column
Chunk
Col2A
Column
Chunk
Col2B
Page
Header
Column Chunk Column 3
Page Header
Page Page Page
- Metadata
- R Values
- D Values
- ENCODED DATA
Slicing and Dicing Within A Compressed File
File Metadata
Row
Group Row Group
Column
Chunk
Col1
Column
Chunk
Col2A
Column
Chunk
Col2B
Page
Header
Column Chunk Column 3
Page Header
Page Page Page
- Metadata
- R Values
- D Values
- ENCODED DATA
Slicing and Dicing Within A Compressed File
File Metadata
Row
Group Row Group
Column
Chunk
Col1
Column
Chunk
Col2A
Column
Chunk
Col2B
Page
Header
Column Chunk Column 3
Page Header
Page Page Page
- Metadata
- R Values
- D Values
- ENCODED DATA
Slicing and Dicing Within A Compressed File
File Metadata
Row
Group Row Group
Column
Chunk
Col1
Column
Chunk
Col2A
Column
Chunk
Col2B
Page
Header
Column Chunk Column 3
Page Header
Page Page Page
- Metadata
- R Values
- D Values
- ENCODED DATA
Slicing and Dicing Within A Compressed File
File Metadata
Row
Group Row Group
Column
Chunk
Col1
Column
Chunk
Col2A
Column
Chunk
Col2B
Page
Header
Column Chunk Column 3
Page Header
Page Page Page
- Metadata
- R Values
- D Values
- ENCODED DATA
Slicing and Dicing Within A Compressed File
File Metadata
Row
Group Row Group
Column
Chunk
Col1
Column
Chunk
Col2A
Column
Chunk
Col2B
Page
Header
Column Chunk Column 3
Page Header
Page Page Page
- Metadata
- R Values
- D Values
- ENCODED DATA
Format Spec
See the format spec for more detail:
https://github.com/apache/parquet-format
Getting Efficiency With Spark
Partitioning
dataFrame
.write
.partitionBy("Whatever", "Columns", "You", "Want")
.parquet(outputFile)
// For a common example
dataFrame
.write
.partitionBy(”Year", "Month", "Day", "Hour")
.parquet(outputFile)
Spark Filter Pushdown
spark.sql.parquet.filterPushdown à true by default since 1.5.0
For Where Clauses, Having clauses, etc. in SparkSQL, The Data Loading layer
will test the condition before pulling a column chunk into spark memory.
select cs_bill_customer_sk customer_sk, cs_item_sk item_sk
from catalog_sales,date_dim
where cs_sold_date_sk = d_date_sk
and d_month_seq between 1200 and 1200 + 11
Example From: https://developer.ibm.com/hadoop/2016/01/14/5-reasons-to-choose-parquet-for-spark-sql/
Slicing and Dicing Within A Compressed File
File Metadata
Row
Group Row Group
Column
Chunk
Col1
Column
Chunk
Col2A
Column
Chunk
Col2B
Page
Header
Column Chunk Column 3
Page Header
Page Page Page
- Metadata
- R Values
- D Values
- ENCODED DATA
Column chunks contain
metadata with statistics
Physical Plan for Reading CSV
[ Scan
CsvRelation(hdfs://rhel10.cisco.com/user/spark/hadoopds1000g/date_dim/*,false,|,",null,PERMISS
IVE,COMMONS,false,false,StructType(StructField(d_date_sk,IntegerType,false),
StructField(d_date_id,StringType,false), StructField(d_date,StringType,true),
StructField(d_month_seq,LongType,true), StructField(d_week_seq,LongType,true),
StructField(d_quarter_seq,LongType,true), StructField(d_year,LongType,true),
StructField(d_dow,LongType,true), StructField(d_moy,LongType,true),
StructField(d_dom,LongType,true), StructField(d_qoy,LongType,true),
StructField(d_fy_year,LongType,true), StructField(d_fy_quarter_seq,LongType,true),
StructField(d_fy_week_seq,LongType,true), StructField(d_day_name,StringType,true),
StructField(d_quarter_name,StringType,true), StructField(d_holiday,StringType,true),
StructField(d_weekend,StringType,true), StructField(d_following_holiday,StringType,true),
StructField(d_first_dom,LongType,true), StructField(d_last_dom,LongType,true),
StructField(d_same_day_ly,LongType,true), StructField(d_same_day_lq,LongType,true),
StructField(d_current_day,StringType,true), StructField(d_current_week,StringType,true),
StructField(d_current_month,StringType,true), StructField(d_current_quarter,StringType,true),
StructField(d_current_year,StringType,true)))[d_date_sk#141,d_date_id#142,d_date#143,d_month_s
eq#144L,d_week_seq#145L,d_quarter_seq#146L,d_year#147L,d_dow#148L,d_moy#149L,d_dom#150L,d_qoy#
151L,d_fy_year#152L,d_fy_quarter_seq#153L,d_fy_week_seq#154L,d_day_name#155,d_quarter_name#156
,d_holiday#157,d_weekend#158,d_following_holiday#159,d_first_dom#160L,d_last_dom#161L,d_same_d
ay_ly#162L,d_same_day_lq#163L,d_current_day#164,d_current_week#165,d_current_month#166,d_curre
nt_quarter#167,d_current_year#168]]
Example From: https://developer.ibm.com/hadoop/2016/01/14/5-reasons-to-choose-parquet-for-spark-sql/
Physical Plan For Reading Parquet
+- Scan ParquetRelation[d_date_sk#141,d_month_seq#144L] InputPaths:
hdfs://rhel10.cisco.com/user/spark/hadoopds1tbparquet/date_dim/_SUCCESS,
hdfs://rhel10.cisco.com/user/spark/hadoopds1tbparquet/date_dim/_common_metadata
, hdfs://rhel10.cisco.com/user/spark/hadoopds1tbparquet/date_dim/_metadata,
hdfs://rhel10.cisco.com/user/spark/hadoopds1tbparquet/date_dim/part-r-00000-
4d205b7e-b21d-4e8b-81ac-d2a1f3dd3246.gz.parquet,
hdfs://rhel10.cisco.com/user/spark/hadoopds1tbparquet/date_dim/part-r-00001-
4d205b7e-b21d-4e8b-81ac-d2a1f3dd3246.gz.parquet, PushedFilters:
[GreaterThanOrEqual(d_month_seq,1200),
LessThanOrEqual(d_month_seq,1211)]]
Example From: https://developer.ibm.com/hadoop/2016/01/14/5-reasons-to-choose-parquet-for-spark-sql/
Get JUST the Data You Need
• Get just the partitions you need
• Get just the columns you need
• Get just the chunks of the columns that fit your
filter conditions
What’s the Catch?
Limitations, Write Speed, Immutability
Limitations
• Pushdown Filtering doesn’t exactly work with
object stores: AWS S3, etc. No random access
• Pushdown Filtering does not work on nested
columns - SPARK-17636
• Binary vs. String saga – SPARK-17213
Write Speed à Who Cares!!
(In Most Cases)
Write Once
Read Forever
Which case will you optimize for?
Dealing With Immutability
• Write using partitioning
– Reimagine your data as a timeseries
• Combine with a database (i.e. Cassandra)
• Append additional row groups
Parquet in a Streaming Context
Ongoing project In the Watson Data Platform
• Collect until watermark condition is met (time, size, number of
rows, etc.)
• Groom collection
• Write groomed rows to parquet
• Append to existing as additional compressed files
Tuning and Tips for Spark + Parquet
Tuning In Spark
(depending on your version)
• Use s3a if you’re in AWS land
• df.read.option(“mergeSchema”,
“false”).parquet(”s3a://whatever”)
• Coalescing will change the number of compressed files produced
• Make sure your Parquet block size == your HDFS block size
• sparkContext.hadoopConfiguration.set(
"spark.sql.parquet.output.committer.class",
"org.apache.spark.sql.parquet.DirectParquetOutputCommitter”)
Let’s Summarize!
In Summary
Parquet
Usability Good!
Administration None!
Spark Integration FANTASTIC!!
Resource Efficiency WONDERFUL!! (Storage, I/O, Data cardinality)
Scalability FANTASTIC!!
CO$$$$T ¢¢¢
QUERY TIME GOOD!!
In Summary
Parquet is a binary data storage format that,
in combination with Spark, enables fast queries by
getting you just the data you need,
getting it efficiently, and
keeping much of the work out of Spark.
The Extra Slides
More About Those Benchmarks
File Format Query Time (sec) Size (GB)
CSV 2892.3 437.46
Parquet: LZO 50.6 55.6
Parquet: Uncompressed 43.4 138.54
Parquet: GZIP 40.3 36.78
Parquet: Snappy 28.9 54.83
SELECT cacheStatus, bytesSent from ADatasetThatHasToDoWithCDNs
WHERE cacheStatus LIKE 'stale'
AND bytesSent < 500
More About Those Benchmarks
• Wimpy cluster
– 1 master
– 3 workers
– EC2 c4.4xlarge nodes
• All data in HDFS
Parquet vs. ORC
• ORC is columnar and indexed
• ORC does not handle nesting
• Table Scan benchmarks: comparable, ORC
sometimes faster
• Selected Columns Benchmarks: Parquet wins
• Benchmarks outdated
– Old versions of Spark
– Old versions of ORC and Parquet spec
Parquet vs. Avro
• Avro is row-major
• Avro can be fast for table scans, but loses
heavily on column-selection queries
Parquet vs. Kudu
• Parquet is immutable on disk
• Kudu is mutable on disk
• Trade-offs for both: http://www.slideshare.net/HadoopSummit/the-
columnar-era-leveraging-parquet-arrow-and-kudu-for-highperformance-
analytics
Robbie Strickland
VP, Engines & Pipelines, Watson Data Platform
Emily May Curtin
Software Engineer, IBM Spark Technology Center East

More Related Content

What's hot

Top 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applicationsTop 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applications
hadooparchbook
 
The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization OpportunitiesThe Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization Opportunities
Databricks
 
Optimizing Apache Spark SQL Joins
Optimizing Apache Spark SQL JoinsOptimizing Apache Spark SQL Joins
Optimizing Apache Spark SQL Joins
Databricks
 
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang WangApache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Databricks
 
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Databricks
 
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
Radical Speed for SQL Queries on Databricks: Photon Under the HoodRadical Speed for SQL Queries on Databricks: Photon Under the Hood
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
Databricks
 
Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014
Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014
Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014
Julien Le Dem
 
Enabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache SparkEnabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache Spark
Kazuaki Ishizaki
 
Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)
Ryan Blue
 
Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
 Best Practice of Compression/Decompression Codes in Apache Spark with Sophia... Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
Databricks
 
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin HuaiA Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
Databricks
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks
 
How We Optimize Spark SQL Jobs With parallel and sync IO
How We Optimize Spark SQL Jobs With parallel and sync IOHow We Optimize Spark SQL Jobs With parallel and sync IO
How We Optimize Spark SQL Jobs With parallel and sync IO
Databricks
 
Understanding and Improving Code Generation
Understanding and Improving Code GenerationUnderstanding and Improving Code Generation
Understanding and Improving Code Generation
Databricks
 
Top 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark ApplicationsTop 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark Applications
Spark Summit
 
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in SparkSpark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Bo Yang
 
Change Data Feed in Delta
Change Data Feed in DeltaChange Data Feed in Delta
Change Data Feed in Delta
Databricks
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Cloudera, Inc.
 
Common Strategies for Improving Performance on Your Delta Lakehouse
Common Strategies for Improving Performance on Your Delta LakehouseCommon Strategies for Improving Performance on Your Delta Lakehouse
Common Strategies for Improving Performance on Your Delta Lakehouse
Databricks
 
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
Databricks
 

What's hot (20)

Top 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applicationsTop 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applications
 
The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization OpportunitiesThe Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization Opportunities
 
Optimizing Apache Spark SQL Joins
Optimizing Apache Spark SQL JoinsOptimizing Apache Spark SQL Joins
Optimizing Apache Spark SQL Joins
 
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang WangApache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
 
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...
 
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
Radical Speed for SQL Queries on Databricks: Photon Under the HoodRadical Speed for SQL Queries on Databricks: Photon Under the Hood
Radical Speed for SQL Queries on Databricks: Photon Under the Hood
 
Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014
Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014
Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014
 
Enabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache SparkEnabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache Spark
 
Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)
 
Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
 Best Practice of Compression/Decompression Codes in Apache Spark with Sophia... Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...
 
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin HuaiA Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
 
How We Optimize Spark SQL Jobs With parallel and sync IO
How We Optimize Spark SQL Jobs With parallel and sync IOHow We Optimize Spark SQL Jobs With parallel and sync IO
How We Optimize Spark SQL Jobs With parallel and sync IO
 
Understanding and Improving Code Generation
Understanding and Improving Code GenerationUnderstanding and Improving Code Generation
Understanding and Improving Code Generation
 
Top 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark ApplicationsTop 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark Applications
 
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in SparkSpark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
 
Change Data Feed in Delta
Change Data Feed in DeltaChange Data Feed in Delta
Change Data Feed in Delta
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
 
Common Strategies for Improving Performance on Your Delta Lakehouse
Common Strategies for Improving Performance on Your Delta LakehouseCommon Strategies for Improving Performance on Your Delta Lakehouse
Common Strategies for Improving Performance on Your Delta Lakehouse
 
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
 

Viewers also liked

Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...
Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...
Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...
Spark Summit
 
Horizontally Scalable Relational Databases with Spark: Spark Summit East talk...
Horizontally Scalable Relational Databases with Spark: Spark Summit East talk...Horizontally Scalable Relational Databases with Spark: Spark Summit East talk...
Horizontally Scalable Relational Databases with Spark: Spark Summit East talk...
Spark Summit
 
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
Spark Summit
 
New Directions in pySpark for Time Series Analysis: Spark Summit East talk by...
New Directions in pySpark for Time Series Analysis: Spark Summit East talk by...New Directions in pySpark for Time Series Analysis: Spark Summit East talk by...
New Directions in pySpark for Time Series Analysis: Spark Summit East talk by...
Spark Summit
 
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
Spark Summit
 
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Spark Summit
 
Going Real-Time: Creating Frequently-Updating Datasets for Personalization: S...
Going Real-Time: Creating Frequently-Updating Datasets for Personalization: S...Going Real-Time: Creating Frequently-Updating Datasets for Personalization: S...
Going Real-Time: Creating Frequently-Updating Datasets for Personalization: S...
Spark Summit
 
Time Series Analytics with Spark: Spark Summit East talk by Simon Ouellette
Time Series Analytics with Spark: Spark Summit East talk by Simon OuelletteTime Series Analytics with Spark: Spark Summit East talk by Simon Ouellette
Time Series Analytics with Spark: Spark Summit East talk by Simon Ouellette
Spark Summit
 
Improving Python and Spark Performance and Interoperability: Spark Summit Eas...
Improving Python and Spark Performance and Interoperability: Spark Summit Eas...Improving Python and Spark Performance and Interoperability: Spark Summit Eas...
Improving Python and Spark Performance and Interoperability: Spark Summit Eas...
Spark Summit
 
Analysis Andromeda Galaxy Data Using Spark: Spark Summit East Talk by Jose Na...
Analysis Andromeda Galaxy Data Using Spark: Spark Summit East Talk by Jose Na...Analysis Andromeda Galaxy Data Using Spark: Spark Summit East Talk by Jose Na...
Analysis Andromeda Galaxy Data Using Spark: Spark Summit East Talk by Jose Na...
Spark Summit
 
BigDL: A Distributed Deep Learning Library on Spark: Spark Summit East talk b...
BigDL: A Distributed Deep Learning Library on Spark: Spark Summit East talk b...BigDL: A Distributed Deep Learning Library on Spark: Spark Summit East talk b...
BigDL: A Distributed Deep Learning Library on Spark: Spark Summit East talk b...
Spark Summit
 
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
Spark Summit
 
Realtime Analytical Query Processing and Predictive Model Building on High Di...
Realtime Analytical Query Processing and Predictive Model Building on High Di...Realtime Analytical Query Processing and Predictive Model Building on High Di...
Realtime Analytical Query Processing and Predictive Model Building on High Di...
Spark Summit
 
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming: Spar...
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming: Spar...Building Realtime Data Pipelines with Kafka Connect and Spark Streaming: Spar...
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming: Spar...
Spark Summit
 
Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...
Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...
Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...
Spark Summit
 
Spark Autotuning: Spark Summit East talk by Lawrence Spracklen
Spark Autotuning: Spark Summit East talk by Lawrence SpracklenSpark Autotuning: Spark Summit East talk by Lawrence Spracklen
Spark Autotuning: Spark Summit East talk by Lawrence Spracklen
Spark Summit
 
Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...
Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...
Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...
Spark Summit
 
Spark SQL: Another 16x Faster After Tungsten: Spark Summit East talk by Brad ...
Spark SQL: Another 16x Faster After Tungsten: Spark Summit East talk by Brad ...Spark SQL: Another 16x Faster After Tungsten: Spark Summit East talk by Brad ...
Spark SQL: Another 16x Faster After Tungsten: Spark Summit East talk by Brad ...
Spark Summit
 
Kerberizing Spark: Spark Summit East talk by Abel Rincon and Jorge Lopez-Malla
Kerberizing Spark: Spark Summit East talk by Abel Rincon and Jorge Lopez-MallaKerberizing Spark: Spark Summit East talk by Abel Rincon and Jorge Lopez-Malla
Kerberizing Spark: Spark Summit East talk by Abel Rincon and Jorge Lopez-Malla
Spark Summit
 
Sparkler—Crawler on Apache Spark: Spark Summit East talk by Karanjeet Singh a...
Sparkler—Crawler on Apache Spark: Spark Summit East talk by Karanjeet Singh a...Sparkler—Crawler on Apache Spark: Spark Summit East talk by Karanjeet Singh a...
Sparkler—Crawler on Apache Spark: Spark Summit East talk by Karanjeet Singh a...
Spark Summit
 

Viewers also liked (20)

Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...
Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...
Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...
 
Horizontally Scalable Relational Databases with Spark: Spark Summit East talk...
Horizontally Scalable Relational Databases with Spark: Spark Summit East talk...Horizontally Scalable Relational Databases with Spark: Spark Summit East talk...
Horizontally Scalable Relational Databases with Spark: Spark Summit East talk...
 
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
 
New Directions in pySpark for Time Series Analysis: Spark Summit East talk by...
New Directions in pySpark for Time Series Analysis: Spark Summit East talk by...New Directions in pySpark for Time Series Analysis: Spark Summit East talk by...
New Directions in pySpark for Time Series Analysis: Spark Summit East talk by...
 
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
 
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
 
Going Real-Time: Creating Frequently-Updating Datasets for Personalization: S...
Going Real-Time: Creating Frequently-Updating Datasets for Personalization: S...Going Real-Time: Creating Frequently-Updating Datasets for Personalization: S...
Going Real-Time: Creating Frequently-Updating Datasets for Personalization: S...
 
Time Series Analytics with Spark: Spark Summit East talk by Simon Ouellette
Time Series Analytics with Spark: Spark Summit East talk by Simon OuelletteTime Series Analytics with Spark: Spark Summit East talk by Simon Ouellette
Time Series Analytics with Spark: Spark Summit East talk by Simon Ouellette
 
Improving Python and Spark Performance and Interoperability: Spark Summit Eas...
Improving Python and Spark Performance and Interoperability: Spark Summit Eas...Improving Python and Spark Performance and Interoperability: Spark Summit Eas...
Improving Python and Spark Performance and Interoperability: Spark Summit Eas...
 
Analysis Andromeda Galaxy Data Using Spark: Spark Summit East Talk by Jose Na...
Analysis Andromeda Galaxy Data Using Spark: Spark Summit East Talk by Jose Na...Analysis Andromeda Galaxy Data Using Spark: Spark Summit East Talk by Jose Na...
Analysis Andromeda Galaxy Data Using Spark: Spark Summit East Talk by Jose Na...
 
BigDL: A Distributed Deep Learning Library on Spark: Spark Summit East talk b...
BigDL: A Distributed Deep Learning Library on Spark: Spark Summit East talk b...BigDL: A Distributed Deep Learning Library on Spark: Spark Summit East talk b...
BigDL: A Distributed Deep Learning Library on Spark: Spark Summit East talk b...
 
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
 
Realtime Analytical Query Processing and Predictive Model Building on High Di...
Realtime Analytical Query Processing and Predictive Model Building on High Di...Realtime Analytical Query Processing and Predictive Model Building on High Di...
Realtime Analytical Query Processing and Predictive Model Building on High Di...
 
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming: Spar...
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming: Spar...Building Realtime Data Pipelines with Kafka Connect and Spark Streaming: Spar...
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming: Spar...
 
Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...
Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...
Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...
 
Spark Autotuning: Spark Summit East talk by Lawrence Spracklen
Spark Autotuning: Spark Summit East talk by Lawrence SpracklenSpark Autotuning: Spark Summit East talk by Lawrence Spracklen
Spark Autotuning: Spark Summit East talk by Lawrence Spracklen
 
Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...
Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...
Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...
 
Spark SQL: Another 16x Faster After Tungsten: Spark Summit East talk by Brad ...
Spark SQL: Another 16x Faster After Tungsten: Spark Summit East talk by Brad ...Spark SQL: Another 16x Faster After Tungsten: Spark Summit East talk by Brad ...
Spark SQL: Another 16x Faster After Tungsten: Spark Summit East talk by Brad ...
 
Kerberizing Spark: Spark Summit East talk by Abel Rincon and Jorge Lopez-Malla
Kerberizing Spark: Spark Summit East talk by Abel Rincon and Jorge Lopez-MallaKerberizing Spark: Spark Summit East talk by Abel Rincon and Jorge Lopez-Malla
Kerberizing Spark: Spark Summit East talk by Abel Rincon and Jorge Lopez-Malla
 
Sparkler—Crawler on Apache Spark: Spark Summit East talk by Karanjeet Singh a...
Sparkler—Crawler on Apache Spark: Spark Summit East talk by Karanjeet Singh a...Sparkler—Crawler on Apache Spark: Spark Summit East talk by Karanjeet Singh a...
Sparkler—Crawler on Apache Spark: Spark Summit East talk by Karanjeet Singh a...
 

Similar to Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie Strickland

Intro to Cassandra
Intro to CassandraIntro to Cassandra
Intro to Cassandra
DataStax Academy
 
Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra
Matthias Niehoff
 
Advanced Apache Spark Meetup: How Spark Beat Hadoop @ 100 TB Daytona GraySor...
Advanced Apache Spark Meetup:  How Spark Beat Hadoop @ 100 TB Daytona GraySor...Advanced Apache Spark Meetup:  How Spark Beat Hadoop @ 100 TB Daytona GraySor...
Advanced Apache Spark Meetup: How Spark Beat Hadoop @ 100 TB Daytona GraySor...
Chris Fregly
 
Advanced Apache Spark Meetup Spark SQL + DataFrames + Catalyst Optimizer + Da...
Advanced Apache Spark Meetup Spark SQL + DataFrames + Catalyst Optimizer + Da...Advanced Apache Spark Meetup Spark SQL + DataFrames + Catalyst Optimizer + Da...
Advanced Apache Spark Meetup Spark SQL + DataFrames + Catalyst Optimizer + Da...
Chris Fregly
 
Spark cassandra connector.API, Best Practices and Use-Cases
Spark cassandra connector.API, Best Practices and Use-CasesSpark cassandra connector.API, Best Practices and Use-Cases
Spark cassandra connector.API, Best Practices and Use-Cases
Duyhai Doan
 
DRUG - RDSTK Talk
DRUG - RDSTK TalkDRUG - RDSTK Talk
DRUG - RDSTK Talk
rtelmore
 
Under The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database ArchitectureUnder The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database Architecture
ScyllaDB
 
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Databricks
 
Hypertable - massively scalable nosql database
Hypertable - massively scalable nosql databaseHypertable - massively scalable nosql database
Hypertable - massively scalable nosql database
bigdatagurus_meetup
 
Enter the Snake Pit for Fast and Easy Spark
Enter the Snake Pit for Fast and Easy SparkEnter the Snake Pit for Fast and Easy Spark
Enter the Snake Pit for Fast and Easy Spark
Jon Haddad
 
23 October 2013 - AWS 201 - A Walk through the AWS Cloud: Introduction to Ama...
23 October 2013 - AWS 201 - A Walk through the AWS Cloud: Introduction to Ama...23 October 2013 - AWS 201 - A Walk through the AWS Cloud: Introduction to Ama...
23 October 2013 - AWS 201 - A Walk through the AWS Cloud: Introduction to Ama...
Amazon Web Services
 
Apache Spark Best Practices Meetup Talk
Apache Spark Best Practices Meetup TalkApache Spark Best Practices Meetup Talk
Apache Spark Best Practices Meetup Talk
Eren Avşaroğulları
 
Functional Comparison and Performance Evaluation of Streaming Frameworks
Functional Comparison and Performance Evaluation of Streaming FrameworksFunctional Comparison and Performance Evaluation of Streaming Frameworks
Functional Comparison and Performance Evaluation of Streaming Frameworks
Huafeng Wang
 
Scylla Summit 2017: Intel Optane SSDs as the New Accelerator in Your Data Center
Scylla Summit 2017: Intel Optane SSDs as the New Accelerator in Your Data CenterScylla Summit 2017: Intel Optane SSDs as the New Accelerator in Your Data Center
Scylla Summit 2017: Intel Optane SSDs as the New Accelerator in Your Data Center
ScyllaDB
 
Rapid Response: Debugging and Profiling to the Rescue
Rapid Response: Debugging and Profiling to the RescueRapid Response: Debugging and Profiling to the Rescue
Rapid Response: Debugging and Profiling to the Rescue
Eric Kavanagh
 
Ben Coverston - The Apache Cassandra Project
Ben Coverston - The Apache Cassandra ProjectBen Coverston - The Apache Cassandra Project
Ben Coverston - The Apache Cassandra Project
Morningstar Tech Talks
 
Thinking in documents
Thinking in documentsThinking in documents
Thinking in documents
César Rodas
 
Bids talk 9.18
Bids talk 9.18Bids talk 9.18
Bids talk 9.18
Travis Oliphant
 
Osd ctw spark
Osd ctw sparkOsd ctw spark
Osd ctw spark
Wisely chen
 
Jsonsaga 100605143125-phpapp02
Jsonsaga 100605143125-phpapp02Jsonsaga 100605143125-phpapp02
Jsonsaga 100605143125-phpapp02
Ramamohan Chokkam
 

Similar to Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie Strickland (20)

Intro to Cassandra
Intro to CassandraIntro to Cassandra
Intro to Cassandra
 
Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra
 
Advanced Apache Spark Meetup: How Spark Beat Hadoop @ 100 TB Daytona GraySor...
Advanced Apache Spark Meetup:  How Spark Beat Hadoop @ 100 TB Daytona GraySor...Advanced Apache Spark Meetup:  How Spark Beat Hadoop @ 100 TB Daytona GraySor...
Advanced Apache Spark Meetup: How Spark Beat Hadoop @ 100 TB Daytona GraySor...
 
Advanced Apache Spark Meetup Spark SQL + DataFrames + Catalyst Optimizer + Da...
Advanced Apache Spark Meetup Spark SQL + DataFrames + Catalyst Optimizer + Da...Advanced Apache Spark Meetup Spark SQL + DataFrames + Catalyst Optimizer + Da...
Advanced Apache Spark Meetup Spark SQL + DataFrames + Catalyst Optimizer + Da...
 
Spark cassandra connector.API, Best Practices and Use-Cases
Spark cassandra connector.API, Best Practices and Use-CasesSpark cassandra connector.API, Best Practices and Use-Cases
Spark cassandra connector.API, Best Practices and Use-Cases
 
DRUG - RDSTK Talk
DRUG - RDSTK TalkDRUG - RDSTK Talk
DRUG - RDSTK Talk
 
Under The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database ArchitectureUnder The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database Architecture
 
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
 
Hypertable - massively scalable nosql database
Hypertable - massively scalable nosql databaseHypertable - massively scalable nosql database
Hypertable - massively scalable nosql database
 
Enter the Snake Pit for Fast and Easy Spark
Enter the Snake Pit for Fast and Easy SparkEnter the Snake Pit for Fast and Easy Spark
Enter the Snake Pit for Fast and Easy Spark
 
23 October 2013 - AWS 201 - A Walk through the AWS Cloud: Introduction to Ama...
23 October 2013 - AWS 201 - A Walk through the AWS Cloud: Introduction to Ama...23 October 2013 - AWS 201 - A Walk through the AWS Cloud: Introduction to Ama...
23 October 2013 - AWS 201 - A Walk through the AWS Cloud: Introduction to Ama...
 
Apache Spark Best Practices Meetup Talk
Apache Spark Best Practices Meetup TalkApache Spark Best Practices Meetup Talk
Apache Spark Best Practices Meetup Talk
 
Functional Comparison and Performance Evaluation of Streaming Frameworks
Functional Comparison and Performance Evaluation of Streaming FrameworksFunctional Comparison and Performance Evaluation of Streaming Frameworks
Functional Comparison and Performance Evaluation of Streaming Frameworks
 
Scylla Summit 2017: Intel Optane SSDs as the New Accelerator in Your Data Center
Scylla Summit 2017: Intel Optane SSDs as the New Accelerator in Your Data CenterScylla Summit 2017: Intel Optane SSDs as the New Accelerator in Your Data Center
Scylla Summit 2017: Intel Optane SSDs as the New Accelerator in Your Data Center
 
Rapid Response: Debugging and Profiling to the Rescue
Rapid Response: Debugging and Profiling to the RescueRapid Response: Debugging and Profiling to the Rescue
Rapid Response: Debugging and Profiling to the Rescue
 
Ben Coverston - The Apache Cassandra Project
Ben Coverston - The Apache Cassandra ProjectBen Coverston - The Apache Cassandra Project
Ben Coverston - The Apache Cassandra Project
 
Thinking in documents
Thinking in documentsThinking in documents
Thinking in documents
 
Bids talk 9.18
Bids talk 9.18Bids talk 9.18
Bids talk 9.18
 
Osd ctw spark
Osd ctw sparkOsd ctw spark
Osd ctw spark
 
Jsonsaga 100605143125-phpapp02
Jsonsaga 100605143125-phpapp02Jsonsaga 100605143125-phpapp02
Jsonsaga 100605143125-phpapp02
 

More from Spark Summit

FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
Spark Summit
 
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
Spark Summit
 
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Apache Spark Structured Streaming Helps Smart Manufacturing with  Xiaochang WuApache Spark Structured Streaming Helps Smart Manufacturing with  Xiaochang Wu
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Spark Summit
 
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Improving Traffic Prediction Using Weather Data  with Ramya RaghavendraImproving Traffic Prediction Using Weather Data  with Ramya Raghavendra
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Spark Summit
 
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
Spark Summit
 
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
Spark Summit
 
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim DowlingApache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Spark Summit
 
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim DowlingApache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Spark Summit
 
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
Spark Summit
 
Next CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub WozniakNext CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub Wozniak
Spark Summit
 
Powering a Startup with Apache Spark with Kevin Kim
Powering a Startup with Apache Spark with Kevin KimPowering a Startup with Apache Spark with Kevin Kim
Powering a Startup with Apache Spark with Kevin Kim
Spark Summit
 
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Improving Traffic Prediction Using Weather Datawith Ramya RaghavendraImproving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Spark Summit
 
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Spark Summit
 
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...How Nielsen Utilized Databricks for Large-Scale Research and Development with...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Spark Summit
 
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spark Summit
 
Goal Based Data Production with Sim Simeonov
Goal Based Data Production with Sim SimeonovGoal Based Data Production with Sim Simeonov
Goal Based Data Production with Sim Simeonov
Spark Summit
 
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Spark Summit
 
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir VolkGetting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Spark Summit
 
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Spark Summit
 
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
Spark Summit
 

More from Spark Summit (20)

FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
 
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
 
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Apache Spark Structured Streaming Helps Smart Manufacturing with  Xiaochang WuApache Spark Structured Streaming Helps Smart Manufacturing with  Xiaochang Wu
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
 
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Improving Traffic Prediction Using Weather Data  with Ramya RaghavendraImproving Traffic Prediction Using Weather Data  with Ramya Raghavendra
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
 
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
 
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
 
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim DowlingApache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
 
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim DowlingApache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
 
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
 
Next CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub WozniakNext CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub Wozniak
 
Powering a Startup with Apache Spark with Kevin Kim
Powering a Startup with Apache Spark with Kevin KimPowering a Startup with Apache Spark with Kevin Kim
Powering a Startup with Apache Spark with Kevin Kim
 
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Improving Traffic Prediction Using Weather Datawith Ramya RaghavendraImproving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
 
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
 
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...How Nielsen Utilized Databricks for Large-Scale Research and Development with...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
 
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
 
Goal Based Data Production with Sim Simeonov
Goal Based Data Production with Sim SimeonovGoal Based Data Production with Sim Simeonov
Goal Based Data Production with Sim Simeonov
 
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
 
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir VolkGetting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir Volk
 
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
 
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
 

Recently uploaded

一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
actyx
 
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call GirlCall Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
sapna sharmap11
 
Drownings spike from May to August in children
Drownings spike from May to August in childrenDrownings spike from May to August in children
Drownings spike from May to August in children
Bisnar Chase Personal Injury Attorneys
 
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
Rebecca Bilbro
 
[VCOSA] Monthly Report - Cotton & Yarn Statistics May 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics May 2024[VCOSA] Monthly Report - Cotton & Yarn Statistics May 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics May 2024
Vietnam Cotton & Spinning Association
 
06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus
Timothy Spann
 
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
hqfek
 
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call GirlCall Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
sapna sharmap11
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
bmucuha
 
How To Control IO Usage using Resource Manager
How To Control IO Usage using Resource ManagerHow To Control IO Usage using Resource Manager
How To Control IO Usage using Resource Manager
Alireza Kamrani
 
Sid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.pptSid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.ppt
ArshadAyub49
 
一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
一比一原版卡尔加里大学毕业证(uc毕业证)如何办理一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
oaxefes
 
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance PaymentCall Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
prijesh mathew
 
Econ3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdfEcon3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdf
blueshagoo1
 
A gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented GenerationA gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented Generation
dataschool1
 
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
Timothy Spann
 
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdfNamma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
22ad0301
 
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
aguty
 
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
Vietnam Cotton & Spinning Association
 
Senior Engineering Sample EM DOE - Sheet1.pdf
Senior Engineering Sample EM DOE  - Sheet1.pdfSenior Engineering Sample EM DOE  - Sheet1.pdf
Senior Engineering Sample EM DOE - Sheet1.pdf
Vineet
 

Recently uploaded (20)

一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
一比一原版斯威本理工大学毕业证(swinburne毕业证)如何办理
 
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call GirlCall Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
 
Drownings spike from May to August in children
Drownings spike from May to August in childrenDrownings spike from May to August in children
Drownings spike from May to August in children
 
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
 
[VCOSA] Monthly Report - Cotton & Yarn Statistics May 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics May 2024[VCOSA] Monthly Report - Cotton & Yarn Statistics May 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics May 2024
 
06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus
 
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
 
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
Call Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call GirlCall Girls Hyderabad  (india) ☎️ +91-7426014248 Hyderabad  Call Girl
Call Girls Hyderabad (india) ☎️ +91-7426014248 Hyderabad Call Girl
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 
How To Control IO Usage using Resource Manager
How To Control IO Usage using Resource ManagerHow To Control IO Usage using Resource Manager
How To Control IO Usage using Resource Manager
 
Sid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.pptSid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.ppt
 
一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
一比一原版卡尔加里大学毕业证(uc毕业证)如何办理一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
一比一原版卡尔加里大学毕业证(uc毕业证)如何办理
 
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance PaymentCall Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
Call Girls Hyderabad ❤️ 7339748667 ❤️ With No Advance Payment
 
Econ3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdfEcon3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdf
 
A gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented GenerationA gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented Generation
 
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
 
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdfNamma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
 
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
 
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
 
Senior Engineering Sample EM DOE - Sheet1.pdf
Senior Engineering Sample EM DOE  - Sheet1.pdfSenior Engineering Sample EM DOE  - Sheet1.pdf
Senior Engineering Sample EM DOE - Sheet1.pdf
 

Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie Strickland

  • 1. Spark + Parquet in Depth Robbie Strickland VP, Engines & Pipelines, Watson Data Platform @rs_atl Emily May Curtin Software Engineer, IBM Spark Technology Center @emilymaycurtin
  • 4. Outline • Why Parquet • Parquet by example • How Parquet works • How Spark squeezes out efficiency • What’s the catch • Tuning tips for Spark + Parquet
  • 5.
  • 6.
  • 7. Goals for Data Lake Storage • Good Usability – Easy to backup – Minimal learning curve – Easy integration with existing tools • Resource efficient – Disk space – Disk I/O Time – Network I/O • AFFORDABLE • CA$$$H MONEY • DEVELOPER HOURS à $$$ • COMPUTE CYCLES à $$$ •FAST QUERIES
  • 8. Little Costs Matter at Actual Scale “Very Large Dataset” Weather-Scale Data
  • 9. Disk and Network I/O Hurt Action Computer Time “Human Scale” Time 1 CPU cycle 0.3 ns 1 s Level 1 cache access 0.9 ns 3 s Level 2 cache access 2.8 ns 9 s Level 3 cache access 12.9 ns 43 s Main memory access 120 ns 6 min Solid-state disk I/O 50-150 μs 2-6 days Rotational disk I/O 1-10 ms 1-12 months Internet: SF to NYC 40 ms 4 years Internet: SF to UK 81 ms 8 years Internet: SF to Australia 183 ms 19 years Source: Systems Performance: Enterprise and the Cloud by Brendan Gregg via CodingHorror.com “The Infinite Space Between Words”
  • 10. Options For Multi-PB Data Lake Storage Files Compressed Files Databases Usability Great! Great! OK to BAD (not as easy as a file!) Administration None! None! LOTS Spark Integration Great! Great! Varies Resource Efficiency BAD (Big storage, heavy I/O) OK… (Less storage) BAD (Requires storage AND CPU) Scalability Good-ish Good-ish BAD (For multi-petabyte!) CO$$$$T OK… OK… TERRIBLE QUERY TIME TERRIBLE BAD Good!
  • 11. CSV vs. Parquet Column Selection Query 2892.3 50.6 43.4 40.3 28.90 500 1000 1500 2000 2500 3000 CSV Parquet: LZO Parquet: Uncompressed Parquet: GZIP Parquet: Snappy Query Time (seconds) SELECT cacheStatus, bytesSent from ADatasetThatHasToDoWithCDNs WHERE cacheStatus LIKE 'stale' AND bytesSent < 500
  • 12. CSV vs. Parquet Table Scan Query 2059 50 49.1 44.2 39.60 500 1000 1500 2000 CSV Parquet: LZO Parquet: Snappy Parquet: Uncompressed Parquet: GZIP Query Time (seconds) SELECT * from ADatasetThatHasToDoWithCDNs WHERE cacheStatus LIKE 'stale' AND bytesSent < 500
  • 13. Parquet Format “Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.” • Binary Format • API for JVM/Hadoop & C++ • Columnar • Encoded • Compressed • Machine-Friendly
  • 15. Very Important Dataset Title Released Label PeakChart.UK Certification.BVMI Certification.RIAA (omitted for space…) Led Zeppelin 01/12/1969 Atlantic 6 8x Platinum … Led Zeppelin II 10/22/1969 Atlantic 1 Platinum Diamond … Led Zeppelin III 10/05/1970 Atlantic 1 Gold 6x Platinum … Led Zeppelin IV 11/08/1971 Atlantic 1 3x Gold Diamond … Houses of the Holy 03/28/1973 Atlantic 1 Gold Diamond … Physical Graffiti 02/24/1975 Swan Song 1 Gold Diamond … Presence 03/31/1976 Swan Song 1 3x Platinum … In Through The Out Door 08/15/1979 Swan Song 1 6x Platinum … Coda 11/19/1982 Swan Song 4 Platinum …
  • 16. One Row, Two Different Ways { "Title" : "Led Zeppelin IV", "Released" : "11/8/1971", "Label" : "Atlantic", "PeakChart.UK" : 1, "PeakChart.AUS" : 2, "PeakChart.US" : 2, "Certification.ARIA" : "9x Platinum", "Certification.BPI" : "6x Platinum", "Certification.BVMI" : "3x Gold", "Certification.CRIA" : "2x Diamond", "Certification.IFPI" : "2x Platinum", "Certification.NVPI" : "Platinum", "Certification.RIAA" : "Diamond", "Certification.SNEP" : "2x Platinum" } { "TITLE": "LED ZEPPELIN IV", "RELEASED": "11/8/1971", "LABEL": "ATLANTIC", "PEAKCHART": { "UK": 1, "AUS": 2, "US": 2 }, "CERTIFICATION": { "ARIA": "9X PLATINUM", "BPI": "6X PLATINUM", "BVMI": "3X GOLD", "CRIA": "2X DIAMOND", "IFPI": "2X PLATINUM", "NVPI": "PLATINUM", "RIAA": "DIAMOND", "SNEP": "2X PLATINUM“ } }
  • 17. The Flat Schema Data Title Released Label PeakChart.UK PeakChart.AUS PeakChart.US PeakChart.Mars Certification.ARIA Certification.BPI Certification.BVMI Certification.CRIA Certification.IFPI Certification.NVPI Certification.RIAA Certification.SNEP Led Zeppelin 01/12/1969 Atlantic 6 9 10 2x Platinum 2x Platinum Diamond Gold Gold 8x Platinum Gold Led Zeppelin II 10/22/1969 Atlantic 1 1 1 4x Platinum 4x Platinum Platinum 9x Platinum Gold Diamond Platinum Led Zeppelin III 10/5/1970 Atlantic 1 1 1 Platinum Gold 3x Platinum Gold Gold 6x Platinum Platinum Led Zeppelin IV 11/8/1971 Atlantic 1 2 2 9x Platinum 6x Platinum 3x Gold 2x Diamond 2x Platinum Platinum Diamond 2x Platinum Houses of the Holy 03/28/1973 Atlantic 1 1 1 Platinum Gold Diamond Gold Physical Graffiti 02/24/1975 Swan Song 1 1 1 3x Platinum 2x Platinum Gold Diamond Gold Presence 03/31/1976 Swan Song 1 4 1 Platinum 3x Platinum In Through The Out Door 08/15/1979 Swan Song 1 3 1 2x Platinum Platinum 6x Platinum Coda 11/19/1982 Swan Song 4 9 6 Silver Platinum
  • 18. The Nested Schema Data {"Title":"Led Zeppelin", "Released":"01/12/1969", "Label":"Atlantic", "PeakChart":{"UK":6, "AUS":9, "US":10}, "Certification":{"ARIA":"2x Platinum", "BPI":"2x Platinum", "CRIA":"Diamond", "IFPI":"Gold", "NVPI":"Gold", "RIAA":"8x Platinum", "SNEP":"Gold"}} {"Title":"Led Zeppelin II", "Released":"10/22/1969", "Label":"Atlantic", "PeakChart":{"UK":1, "AUS":1, "US":1}, "Certification":{"ARIA":"4x Platinum", "BPI":"4x Platinum", "BVMI":"Platinum", "CRIA":"9x Platinum", "IFPI":"Gold", "RIAA":"Diamond", "SNEP":"Platinum"}} {"Title":"Led Zeppelin III", "Released":"10/5/1970", "Label":"Atlantic", "PeakChart":{"UK":1, "AUS":1, "US":1}, "Certification":{"BPI":"Platinum", "BVMI":"Gold", "CRIA":"3x Platinum", "IFPI":"Gold", "NVPI":"Gold", "RIAA":"6x Platinum", "SNEP":"Platinum"}} {"Title":"Led Zeppelin IV", "Released":"11/8/1971", "Label":"Atlantic", "PeakChart":{"UK":1, "AUS":2, "US":2}, "Certification":{"ARIA":"9x Platinum", "BPI":"6x Platinum", "BVMI":"3x Gold", "CRIA":"2x Diamond", "IFPI":"2x Platinum", "NVPI":"Platinum", "RIAA":"Diamond", "SNEP":"2x Platinum"}} {"Title":"Houses of the Holy", "Released":"03/28/1973", "Label":"Atlantic", "PeakChart":{"UK":1, "AUS":1, "US":1}, "Certification":{"BPI":"Platinum", "BVMI":"Gold", "RIAA":"Diamond", "SNEP":"Gold"}} {"Title":"Physical Graffiti", "Released":"02/24/1975", "Label":"Swan Song", "PeakChart":{"UK":1, "AUS":1, "US":1}, "Certification":{"ARIA":"3x Platinum", "BPI":"2x Platinum", "BVMI":"Gold", "RIAA":"Diamond", "SNEP":"Gold"}} {"Title":"Presence", "Released":"03/31/1976", "Label":"Swan Song", "PeakChart":{"UK":1, "AUS":4, "US":1}, "Certification":{"BPI":"Platinum", "RIAA":"3x Platinum"}} {"Title":"In Through The Out Door", "Released":"08/15/1979", "Label":"Swan Song", "PeakChart":{"UK":1, "AUS":3, "US":1}, "Certification":{"ARIA":"2x Platinum", "BPI":"Platinum", "RIAA":"6x Platinum"}} {"Title":"Coda", "Released":"11/19/1982", "Label":"Swan Song", "PeakChart":{"UK":4, "AUS":9, "US":6}, "Certification":{"BPI":"Silver", "RIAA":"Platinum"}}
  • 19. Parquet By Example Writing Parquet Using Spark
  • 20. Writing To Parquet: Flat Schema val flatDF = spark .read.option("delimiter", "t") .option("header", "true").csv(flatInput) .rdd .map(r => transformRow(r)) .toDF flatDF.write .option("compression", "snappy") .parquet(flatOutput)
  • 21. Writing To Parquet: Flat Schema /*Oh crap, the Ints are gonna get pulled in as Strings unless we transform*/ case class LedZeppelinFlat( Title: Option[String], Released: Option[String], Label: Option[String], UK: Option[Int], AUS: Option[Int], US: Option[Int], ARIA: Option[String], BPI: Option[String], BVMI: Option[String], CRIA: Option[String], IFPI: Option[String], NVPI: Option[String], RIAA: Option[String], SNEP: Option[String] )
  • 22. Writing To Parquet: Flat Schema def transformRow(r: Row): LedZeppelinFlat = { def getStr(r: Row, i: Int) = if(!r.isNullAt(i)) Some(r.getString(i)) else None def getInt(r: Row, i: Int) = if(!r.isNullAt(i)) Some(r.getInt(i)) else None LedZeppelinFlat( getStr(r, 0), getStr(r, 1), getStr(r, 2), getInt(r, 3), getInt(r, 4), getInt(r, 5), getStr(r, 7), getStr(r, 8), getStr(r, 9), getStr(r, 10), getStr(r, 11), getStr(r, 12), getStr(r, 13), getStr(r, 14) ) }
  • 23. Writing To Parquet: Flat Schema val outDF = spark .read.option("delimiter", "t") .option("header", "true").csv(flatInput) .rdd .map(r => transformRow(r)) .toDF outDF.write .option("compression", "snappy") .parquet(flatOutput)
  • 24. Writing To Parquet: Flat Schema
  • 25. Writing To Parquet: Flat Schema… In Java
  • 26. Writing To Parquet: Flat Schema… With MapReduce
  • 27. Writing To Parquet: Nested Schema val nestedDF = spark.read.json(nestedInput) nestedDF.write .option("compression", "snappy") .parquet(nestedOutput)
  • 28. Writing To Parquet: Nested Schema
  • 29. Parquet By Example Let’s See An Example!
  • 30. Parquet Schema Two Different Ways Nested Schema Title: OPTIONAL BINARY O:UTF8 R:0 D:1 Released: OPTIONAL BINARY O:UTF8 R:0 D:1 Label: OPTIONAL BINARY O:UTF8 R:0 D:1 PeakChart: OPTIONAL F:3 .AUS: OPTIONAL INT64 R:0 D:2 .UK: OPTIONAL INT64 R:0 D:2 .US: OPTIONAL INT64 R:0 D:2 Certification: OPTIONAL F:8 .ARIA: OPTIONAL BINARY O:UTF8 R:0 D:2 .BPI: OPTIONAL BINARY O:UTF8 R:0 D:2 .BVMI: OPTIONAL BINARY O:UTF8 R:0 D:2 .CRIA: OPTIONAL BINARY O:UTF8 R:0 D:2 .IFPI: OPTIONAL BINARY O:UTF8 R:0 D:2 .NVPI: OPTIONAL BINARY O:UTF8 R:0 D:2 .RIAA: OPTIONAL BINARY O:UTF8 R:0 D:2 .SNEP: OPTIONAL BINARY O:UTF8 R:0 D:2 FLAT SCHEMA TITLE: OPTIONAL BINARY O:UTF8 R:0 D:1 RELEASED: OPTIONAL BINARY O:UTF8 R:0 D:1 LABEL: OPTIONAL BINARY O:UTF8 R:0 D:1 PEAKCHART.UK: REQUIRED INT32 R:0 D:0 PEAKCHART.AUS: REQUIRED INT32 R:0 D:0 PEAKCHART.US: REQUIRED INT32 R:0 D:0 CERTIFICATION.ARIA: OPTIONAL BINARY O:UTF8 R:0 D:1 CERTIFICATION.BPI: OPTIONAL BINARY O:UTF8 R:0 D:1 CERTIFICATION.BVMI: OPTIONAL BINARY O:UTF8 R:0 D:1 CERTIFICATION.CRIA: OPTIONAL BINARY O:UTF8 R:0 D:1 CERTIFICATION.IFPI: OPTIONAL BINARY O:UTF8 R:0 D:1 CERTIFICATION.NVPI: OPTIONAL BINARY O:UTF8 R:0 D:1 CERTIFICATION.RIAA: OPTIONAL BINARY O:UTF8 R:0 D:1 CERTIFICATION.SNEP: OPTIONAL BINARY O:UTF8 R:0 D:1
  • 31. Schema Breakdown COLUMN NAME Title OPTIONAL / REQUIRED / REPEATED OPTIONAL DATA TYPE BINARY ENCODING INFO FOR BINARY 0:UTF8 REPETITION VALUE R:0 DEFINITION VALUE D:0 FLAT SCHEMA TITLE: OPTIONAL BINARY O:UTF8 R:0 D:1 RELEASED: OPTIONAL BINARY O:UTF8 R:0 D:1 LABEL: OPTIONAL BINARY O:UTF8 R:0 D:1 PEAKCHART.UK: REQUIRED INT32 R:0 D:0 . . .
  • 32. Repetition and Definition Levels Source: https://github.com/apache/parquet-mr
  • 33. One Parquet Row, Two Ways Title = Led Zeppelin IV Released = 11/8/1971 Label = Atlantic PeakChart: .AUS = 2 .UK = 1 .US = 2 Certification: .ARIA = 9x Platinum .BPI = 6x Platinum .BVMI = 3x Gold .CRIA = 2x Diamond .IFPI = 2x Platinum .NVPI = Platinum .RIAA = Diamond .SNEP = 2x Platinum TITLE = LED ZEPPELIN IV RELEASED = 11/8/1971 LABEL = ATLANTIC PEAKCHART.UK = 1 PEAKCHART.AUS = 2 PEAKCHART.US = 2 CERTIFICATION.ARIA = 9X PLATINUM CERTIFICATION.BPI = 6X PLATINUM CERTIFICATION.BVMI = 3X GOLD CERTIFICATION.CRIA = 2X DIAMOND CERTIFICATION.IFPI = 2X PLATINUM CERTIFICATION.NVPI = PLATINUM CERTIFICATION.RIAA = DIAMOND CERTIFICATION.SNEP = 2X PLATINUM
  • 34. Parquet By Example Reading and Querying Using Spark
  • 35. Slightly Different Queries // Many ways, this is just one! val flatParquet = "s3a://…../LedZeppelin-FlatSchema.parquet/" val flatdf = spark.read.parquet(flatParquet) flatdf.createOrReplaceTempView("LedZeppelinFlat") val nestedParquet = "s3a://…../LedZeppelin-NestedSchema.parquet/" val nesteddf = spark.read.parquet(nestedParquet) nesteddf.createOrReplaceTempView("LedZeppelinNested") val flatQuery= "select Title, US from LedZeppelinFlat where US = 1" val nestedQuery = "select Title, PeakChart.US from LedZeppelinNested where PeakChart.US = 1" spark.sql(flatQuery) spark.sql(nestedQuery)
  • 38. Parquet Structure In the Filesystem led-zeppelin-albums.parquet/ • _SUCCESS • _common_metadata • _metadata • Year=1969/ – Part-r-00000-6d4d42e2-c13f-4bdf-917d-2152b24a0f24.snappy.parquet – Part-r-00001-6d4d42e2-c13f-4bdf-917d-2152b24a0f24.snappy.parquet – … • Year=1970/ – Part-r-00000-35cb7ef4-6de6-4efa-9bc6-5286de520af7.snappy.parquet – ... • Groups of rows, partitioned by column values, compressed however you like. (GZIP, LZO, Snappy, etc) • In general LZO wins size benchmarks, Snappy good balance between size and CPU intensity.
  • 39. Data In Columns On Disk Led Zeppelin IV 11/08/1971 1 Houses of the Holy 03/28/1973 1 Physical Graffiti 02/24/1975 1 Led Zeppelin IV 11/08/1971 1Houses of the Holy 03/28/1973 1Physical Graffiti 02/24/1975 1 Row-Oriented data on disk Column-Oriented data on disk Title Date Chart
  • 40. Encoding: Incremental Encoding Led_Zeppelin_IV Led_Zeppelin_III Led_Zeppelin_II Led_Zeppelin 0 Led_Zeppelin 12 _II 15 I 14 V 58 bytes* 24 bytes* *not counting delimiters ENCODING 58% Reduction
  • 41. Encoding: Dictionary Encoding ENCODING Atlantic Swan Song Atlantic Atlantic Atlantic Atlantic Atlantic Swan Song Swan Song Swan Song 84 bytes* 0 1 0 0 0 0 0 1 1 1 0 à Atlantic 1 à Swan Song 1.25 bytes + dictionary size ~98% Reduction
  • 42. More Encoding Schemes • Plain (bit-packed, little endian, etc) • Dictionary Encoding • Run Length Encoding/Bit Packing Hybrid • Delta Encoding • Delta-Length Byte Array • Delta Strings (incremental Encoding) See https://github.com/apache/parquet-format/blob/master/Encodings.md for more detail
  • 43. Slicing and Dicing Within A Compressed File File Metadata Row Group Row Group Column Chunk Col1 Column Chunk Col2A Column Chunk Col2B Page Header Column Chunk Column 3 Page Header Page Page Page - Metadata - R Values - D Values - ENCODED DATA
  • 44. Slicing and Dicing Within A Compressed File File Metadata Row Group Row Group Column Chunk Col1 Column Chunk Col2A Column Chunk Col2B Page Header Column Chunk Column 3 Page Header Page Page Page - Metadata - R Values - D Values - ENCODED DATA
  • 45. Slicing and Dicing Within A Compressed File File Metadata Row Group Row Group Column Chunk Col1 Column Chunk Col2A Column Chunk Col2B Page Header Column Chunk Column 3 Page Header Page Page Page - Metadata - R Values - D Values - ENCODED DATA
  • 46. Slicing and Dicing Within A Compressed File File Metadata Row Group Row Group Column Chunk Col1 Column Chunk Col2A Column Chunk Col2B Page Header Column Chunk Column 3 Page Header Page Page Page - Metadata - R Values - D Values - ENCODED DATA
  • 47. Slicing and Dicing Within A Compressed File File Metadata Row Group Row Group Column Chunk Col1 Column Chunk Col2A Column Chunk Col2B Page Header Column Chunk Column 3 Page Header Page Page Page - Metadata - R Values - D Values - ENCODED DATA
  • 48. Slicing and Dicing Within A Compressed File File Metadata Row Group Row Group Column Chunk Col1 Column Chunk Col2A Column Chunk Col2B Page Header Column Chunk Column 3 Page Header Page Page Page - Metadata - R Values - D Values - ENCODED DATA
  • 49. Slicing and Dicing Within A Compressed File File Metadata Row Group Row Group Column Chunk Col1 Column Chunk Col2A Column Chunk Col2B Page Header Column Chunk Column 3 Page Header Page Page Page - Metadata - R Values - D Values - ENCODED DATA
  • 50. Slicing and Dicing Within A Compressed File File Metadata Row Group Row Group Column Chunk Col1 Column Chunk Col2A Column Chunk Col2B Page Header Column Chunk Column 3 Page Header Page Page Page - Metadata - R Values - D Values - ENCODED DATA
  • 51. Format Spec See the format spec for more detail: https://github.com/apache/parquet-format
  • 53. Partitioning dataFrame .write .partitionBy("Whatever", "Columns", "You", "Want") .parquet(outputFile) // For a common example dataFrame .write .partitionBy(”Year", "Month", "Day", "Hour") .parquet(outputFile)
  • 54. Spark Filter Pushdown spark.sql.parquet.filterPushdown à true by default since 1.5.0 For Where Clauses, Having clauses, etc. in SparkSQL, The Data Loading layer will test the condition before pulling a column chunk into spark memory. select cs_bill_customer_sk customer_sk, cs_item_sk item_sk from catalog_sales,date_dim where cs_sold_date_sk = d_date_sk and d_month_seq between 1200 and 1200 + 11 Example From: https://developer.ibm.com/hadoop/2016/01/14/5-reasons-to-choose-parquet-for-spark-sql/
  • 55. Slicing and Dicing Within A Compressed File File Metadata Row Group Row Group Column Chunk Col1 Column Chunk Col2A Column Chunk Col2B Page Header Column Chunk Column 3 Page Header Page Page Page - Metadata - R Values - D Values - ENCODED DATA Column chunks contain metadata with statistics
  • 56. Physical Plan for Reading CSV [ Scan CsvRelation(hdfs://rhel10.cisco.com/user/spark/hadoopds1000g/date_dim/*,false,|,",null,PERMISS IVE,COMMONS,false,false,StructType(StructField(d_date_sk,IntegerType,false), StructField(d_date_id,StringType,false), StructField(d_date,StringType,true), StructField(d_month_seq,LongType,true), StructField(d_week_seq,LongType,true), StructField(d_quarter_seq,LongType,true), StructField(d_year,LongType,true), StructField(d_dow,LongType,true), StructField(d_moy,LongType,true), StructField(d_dom,LongType,true), StructField(d_qoy,LongType,true), StructField(d_fy_year,LongType,true), StructField(d_fy_quarter_seq,LongType,true), StructField(d_fy_week_seq,LongType,true), StructField(d_day_name,StringType,true), StructField(d_quarter_name,StringType,true), StructField(d_holiday,StringType,true), StructField(d_weekend,StringType,true), StructField(d_following_holiday,StringType,true), StructField(d_first_dom,LongType,true), StructField(d_last_dom,LongType,true), StructField(d_same_day_ly,LongType,true), StructField(d_same_day_lq,LongType,true), StructField(d_current_day,StringType,true), StructField(d_current_week,StringType,true), StructField(d_current_month,StringType,true), StructField(d_current_quarter,StringType,true), StructField(d_current_year,StringType,true)))[d_date_sk#141,d_date_id#142,d_date#143,d_month_s eq#144L,d_week_seq#145L,d_quarter_seq#146L,d_year#147L,d_dow#148L,d_moy#149L,d_dom#150L,d_qoy# 151L,d_fy_year#152L,d_fy_quarter_seq#153L,d_fy_week_seq#154L,d_day_name#155,d_quarter_name#156 ,d_holiday#157,d_weekend#158,d_following_holiday#159,d_first_dom#160L,d_last_dom#161L,d_same_d ay_ly#162L,d_same_day_lq#163L,d_current_day#164,d_current_week#165,d_current_month#166,d_curre nt_quarter#167,d_current_year#168]] Example From: https://developer.ibm.com/hadoop/2016/01/14/5-reasons-to-choose-parquet-for-spark-sql/
  • 57. Physical Plan For Reading Parquet +- Scan ParquetRelation[d_date_sk#141,d_month_seq#144L] InputPaths: hdfs://rhel10.cisco.com/user/spark/hadoopds1tbparquet/date_dim/_SUCCESS, hdfs://rhel10.cisco.com/user/spark/hadoopds1tbparquet/date_dim/_common_metadata , hdfs://rhel10.cisco.com/user/spark/hadoopds1tbparquet/date_dim/_metadata, hdfs://rhel10.cisco.com/user/spark/hadoopds1tbparquet/date_dim/part-r-00000- 4d205b7e-b21d-4e8b-81ac-d2a1f3dd3246.gz.parquet, hdfs://rhel10.cisco.com/user/spark/hadoopds1tbparquet/date_dim/part-r-00001- 4d205b7e-b21d-4e8b-81ac-d2a1f3dd3246.gz.parquet, PushedFilters: [GreaterThanOrEqual(d_month_seq,1200), LessThanOrEqual(d_month_seq,1211)]] Example From: https://developer.ibm.com/hadoop/2016/01/14/5-reasons-to-choose-parquet-for-spark-sql/
  • 58. Get JUST the Data You Need • Get just the partitions you need • Get just the columns you need • Get just the chunks of the columns that fit your filter conditions
  • 59. What’s the Catch? Limitations, Write Speed, Immutability
  • 60. Limitations • Pushdown Filtering doesn’t exactly work with object stores: AWS S3, etc. No random access • Pushdown Filtering does not work on nested columns - SPARK-17636 • Binary vs. String saga – SPARK-17213
  • 61. Write Speed à Who Cares!! (In Most Cases) Write Once Read Forever Which case will you optimize for?
  • 62. Dealing With Immutability • Write using partitioning – Reimagine your data as a timeseries • Combine with a database (i.e. Cassandra) • Append additional row groups
  • 63. Parquet in a Streaming Context Ongoing project In the Watson Data Platform • Collect until watermark condition is met (time, size, number of rows, etc.) • Groom collection • Write groomed rows to parquet • Append to existing as additional compressed files
  • 64. Tuning and Tips for Spark + Parquet
  • 65. Tuning In Spark (depending on your version) • Use s3a if you’re in AWS land • df.read.option(“mergeSchema”, “false”).parquet(”s3a://whatever”) • Coalescing will change the number of compressed files produced • Make sure your Parquet block size == your HDFS block size • sparkContext.hadoopConfiguration.set( "spark.sql.parquet.output.committer.class", "org.apache.spark.sql.parquet.DirectParquetOutputCommitter”)
  • 67. In Summary Parquet Usability Good! Administration None! Spark Integration FANTASTIC!! Resource Efficiency WONDERFUL!! (Storage, I/O, Data cardinality) Scalability FANTASTIC!! CO$$$$T ¢¢¢ QUERY TIME GOOD!!
  • 68. In Summary Parquet is a binary data storage format that, in combination with Spark, enables fast queries by getting you just the data you need, getting it efficiently, and keeping much of the work out of Spark.
  • 69.
  • 71. More About Those Benchmarks File Format Query Time (sec) Size (GB) CSV 2892.3 437.46 Parquet: LZO 50.6 55.6 Parquet: Uncompressed 43.4 138.54 Parquet: GZIP 40.3 36.78 Parquet: Snappy 28.9 54.83 SELECT cacheStatus, bytesSent from ADatasetThatHasToDoWithCDNs WHERE cacheStatus LIKE 'stale' AND bytesSent < 500
  • 72. More About Those Benchmarks • Wimpy cluster – 1 master – 3 workers – EC2 c4.4xlarge nodes • All data in HDFS
  • 73. Parquet vs. ORC • ORC is columnar and indexed • ORC does not handle nesting • Table Scan benchmarks: comparable, ORC sometimes faster • Selected Columns Benchmarks: Parquet wins • Benchmarks outdated – Old versions of Spark – Old versions of ORC and Parquet spec
  • 74. Parquet vs. Avro • Avro is row-major • Avro can be fast for table scans, but loses heavily on column-selection queries
  • 75. Parquet vs. Kudu • Parquet is immutable on disk • Kudu is mutable on disk • Trade-offs for both: http://www.slideshare.net/HadoopSummit/the- columnar-era-leveraging-parquet-arrow-and-kudu-for-highperformance- analytics
  • 76. Robbie Strickland VP, Engines & Pipelines, Watson Data Platform Emily May Curtin Software Engineer, IBM Spark Technology Center East