Successfully reported this slideshow.
Your SlideShare is downloading. ×

The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 34 Ad

The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro

Download to read offline

Zstandard is a fast compression algorithm which you can use in Apache Spark in various way. In this talk, I briefly summarized the evolution history of Apache Spark in this area and four main use cases and the benefits and the next steps:
1) ZStandard can optimize Spark local disk IO by compressing shuffle files significantly. This is very useful in K8s environments. It’s beneficial not only when you use `emptyDir` with `memory` medium, but also it maximizes OS cache benefit when you use shared SSDs or container local storage. In Spark 3.2, SPARK-34390 takes advantage of ZStandard buffer pool feature and its performance gain is impressive, too.
2) Event log compression is another area to save your storage cost on the cloud storage like S3 and to improve the usability. SPARK-34503 officially switched the default event log compression codec from LZ4 to Zstandard.
3) Zstandard data file compression can give you more benefits when you use ORC/Parquet files as your input and output. Apache ORC 1.6 supports Zstandardalready and Apache Spark enables it via SPARK-33978. The upcoming Parquet 1.12 will support Zstandard compression.
4) Last, but not least, since Apache Spark 3.0, Zstandard is used to serialize/deserialize MapStatus data instead of Gzip.

There are more community works to utilize Zstandard to improve Spark. For example, Apache Avro community also supports Zstandard and SPARK-34479 aims to support Zstandard in Spark’s avro file format in Spark 3.2.0.

Zstandard is a fast compression algorithm which you can use in Apache Spark in various way. In this talk, I briefly summarized the evolution history of Apache Spark in this area and four main use cases and the benefits and the next steps:
1) ZStandard can optimize Spark local disk IO by compressing shuffle files significantly. This is very useful in K8s environments. It’s beneficial not only when you use `emptyDir` with `memory` medium, but also it maximizes OS cache benefit when you use shared SSDs or container local storage. In Spark 3.2, SPARK-34390 takes advantage of ZStandard buffer pool feature and its performance gain is impressive, too.
2) Event log compression is another area to save your storage cost on the cloud storage like S3 and to improve the usability. SPARK-34503 officially switched the default event log compression codec from LZ4 to Zstandard.
3) Zstandard data file compression can give you more benefits when you use ORC/Parquet files as your input and output. Apache ORC 1.6 supports Zstandardalready and Apache Spark enables it via SPARK-33978. The upcoming Parquet 1.12 will support Zstandard compression.
4) Last, but not least, since Apache Spark 3.0, Zstandard is used to serialize/deserialize MapStatus data instead of Gzip.

There are more community works to utilize Zstandard to improve Spark. For example, Apache Avro community also supports Zstandard and SPARK-34479 aims to support Zstandard in Spark’s avro file format in Spark 3.2.0.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro (20)

Advertisement

More from Databricks (20)

Recently uploaded (20)

Advertisement

The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro

  1. 1. Apple logo is a trademark of Apple Inc. Dongjoon Hyun Pang Wu The Rise of ZStandard DATA+AI Summit 2021 THIS IS NOT A CONTRIBUTION
  2. 2. = This is not a contribution. Who am I Dongjoon Hyun Apache Spark PMC member and Committer Apache ORC PMC member and Committer Apache REEF PMC member and Committer https://github.com/dongjoon-hyun https://www.linkedin.com/in/dongjoon @dongjoonhyun
  3. 3. = This is not a contribution. Who am I Pang Wu Software Engineer @Apple Maps related data pipelines & dev-tools Work closely with Apple’s Spark PMC to deliver new features. https://www.linkedin.com/in/pangwu/
  4. 4. Agenda ZStandard Issues History When / Why / How to Use Limitations Summary
  5. 5. = This is not a contribution. A fast compression algorithm, providing high compression ratios ZStandard (v1.4.9) Tunable with compression levels https://facebook.github.io/zstd/
  6. 6. = This is not a contribution. Requires Hadoop 2.9+ and pre-built with zStandard library Issue 1: Apache Hadoop ZStandardCodec Apache Spark 3.1.1 distribution with Hadoop 3.2 fails in K8s env scala> spark.range(10).write.option("compression", "zstd").parquet("/tmp/p") java.lang.RuntimeException: native zStandard library not available
  7. 7. = This is not a contribution. Requires Hadoop 2.9+ and pre-built with zStandard library Issue 1: Apache Hadoop ZStandardCodec Apache Spark 3.1.1 distribution with Hadoop 3.2 fails in K8s env scala> spark.range(10).write.option("compression", "zstd").parquet("/tmp/p") java.lang.RuntimeException: native zStandard library not available
  8. 8. = This is not a contribution. Requires Hadoop 2.9+ and pre-built with zStandard library Issue 1: Apache Hadoop ZStandardCodec Apache Spark 3.1.1 distribution with Hadoop 3.2 fails in K8s env Use own codec classes 
 by using zstd-jni or aircompressor library scala> spark.range(10).write.option("compression", "zstd").parquet("/tmp/p") java.lang.RuntimeException: native zStandard library not available
  9. 9. = This is not a contribution. Slow compression and decompression speed Issue 2: Buffer management Use RecyclingBufferPool (SPARK-34340/PARQUET-1973/AVRO-3060)
  10. 10. = This is not a contribution. Slow compression and decompression speed Issue 2: Buffer management Use RecyclingBufferPool (SPARK-34340/PARQUET-1973/AVRO-3060) Compression speedup 0x 1x 2x 3x 4x Level 1 Level 2 Level 3 NoPool RecyclingBufferPool https://issues.apache.org/jira/browse/SPARK-34387 Decompression speedup 0x 0.5x 1x 1.5x 2x Level 1 Level 2 Level 3 NoPool RecyclingBufferPool
  11. 11. = This is not a contribution. Own codecs require more memory than other compression algorithms Issue 2: Buffer management (Cont.) `OOMKilled` may happen in K8s environment when we switch to zstd NAME READY STATUS RESTARTS AGE job 1/1 Running 0 16m job-exec-1 0/1 OOMKilled 0 16m job-exec-2 0/1 OOMKilled 0 16m
  12. 12. = This is not a contribution. Own codecs require more memory than other compression algorithms Issue 2: Buffer management (Cont.) `OOMKilled` may happen in K8s environment when we switch to zstd Use ZStdNoFinalizer to improve GC (zstd-jni 1.4.8+) NAME READY STATUS RESTARTS AGE job 1/1 Running 0 16m job-exec-1 0/1 OOMKilled 0 16m job-exec-2 0/1 OOMKilled 0 16m
  13. 13. = This is not a contribution. Different zstd-jni versions in Spark/Parquet/Avro/Kafka are incompatible Issue 3: zstd-jni inconsistency API Incompatibility - https://github.com/luben/zstd-jni/issues/161
  14. 14. = This is not a contribution. Different zstd-jni versions in Spark/Parquet/Avro/Kafka are incompatible Issue 3: zstd-jni inconsistency API Incompatibility - https://github.com/luben/zstd-jni/issues/161
  15. 15. = This is not a contribution. Different zstd-jni versions in Spark/Parquet/Avro/Kafka are incompatible Issue 3: zstd-jni inconsistency API Incompatibility - https://github.com/luben/zstd-jni/issues/161 Performance inconsistency - v1.4.5-7 BufferPool was added as the default - v1.4.5-8 RecyclingBufferPool was added an d
 BufferPool became an interface - v1.4.7+ NoPool is used by default
  16. 16. = This is not a contribution. Different zstd-jni versions in Spark/Parquet/Avro/Kafka are incompatible Issue 3: zstd-jni inconsistency API Incompatibility - https://github.com/luben/zstd-jni/issues/161 Performance inconsistency - v1.4.5-7 BufferPool was added as the default - v1.4.5-8 RecyclingBufferPool was added an d
 BufferPool became an interface - v1.4.7+ NoPool is used by default Upgrade Spark and dependent Apache projects to use zstd-jni 1.4.9-1 
 (SPARK-34670, PARQUET-1994, AVRO-3072, KAFKA-12442)
  17. 17. = This is not a contribution. Apache Spark with ZStandard History v2.3 Add ZStdCompressionCodec SPARK-19112
  18. 18. = This is not a contribution. Apache Spark with ZStandard History v2.3 Add ZStdCompressionCodec v2.4 Add Apache Hadoop 3.1 profile Use Apache Parquet 1.10 with Hadoop ZStandardCodec SPARK-19112 SPARK-23807 SPARK-23972
  19. 19. = This is not a contribution. Apache Spark with ZStandard History v2.3 Add ZStdCompressionCodec v2.4 Add Apache Hadoop 3.1 profile Use Apache Parquet 1.10 with Hadoop ZStandardCodec v3.0 Broadcast MapStatus with ZStdCompressionCodec Split event log compression from IO compression v3.1 Upgrade to Zstd-jni 1.4.8 SPARK-19112 SPARK-23807 SPARK-23972 SPARK-29434 SPARK-28118 SPARK-33843
  20. 20. = This is not a contribution. Apache Parquet/ORC/Avro with ZStandard History (Cont.) Apache Parquet 1.12.0+ - PARQUET-1866: Replace Hadoop ZSTD with JNI-ZSTD - PARQUET-1973: Support ZSTD JNI BufferPool - PARQUET-1994: Upgrade ZSTD JNI to 1.4.9-1 Apache ORC 1.6.0+ - ORC-363: Enable zStandard codec - ORC-395: Support ZSTD in C++ writer/reader Apache Avro 1.10.2+ - AVRO-2195: Add Zstandard Codec - AVRO-3072: Use ZSTD NoFinalizer classes and bump to 1.4.9-1 - AVRO-3060: Support ZSTD level and BufferPool options
  21. 21. = This is not a contribution. Apache Parquet/ORC/Avro with ZStandard History (Cont.) Apache Parquet 1.12.0+ - PARQUET-1866: Replace Hadoop ZSTD with JNI-ZSTD - PARQUET-1973: Support ZSTD JNI BufferPool - PARQUET-1994: Upgrade ZSTD JNI to 1.4.9-1 Apache ORC 1.6.0+ - ORC-363: Enable zStandard codec - ORC-395: Support ZSTD in C++ writer/reader Apache Avro 1.10.2+ - AVRO-2195: Add Zstandard Codec - AVRO-3072: Use ZSTD NoFinalizer classes and bump to 1.4.9-1 - AVRO-3060: Support ZSTD level and BufferPool options SPARK-34651 Improve ZSTD support
  22. 22. Agenda ZStandard Issues History When / Why / How to Use Limitations Summary
  23. 23. Use spark.eventLog.compression.codec=zstd Spark Event Log Event Log Size (TPCDS 3TB) 0 MB 1600 MB 3200 MB TEXT LZ4 ZSTD 17x smaller than TEXT 
 3x smaller than LZ4 spark.eventLog.enabled=true 
 spark.eventLog.compress=true 
 spark.eventLog.compression.codec=zstd
  24. 24. Shuffle IO NAME READY STATUS RESTARTS AGE disk-emptydir 1/1 Running 0 16m disk-emptydir-exec-1 0/1 Evicted 0 16m disk-emptydir-exec-2 0/1 Evicted 0 16m
  25. 25. Use spark.io.compression.codec=zstd Shuffle IO (Cont.) Shuffle Write Size (TPCDS 3TB) 0 TB 3 TB 6 TB 9 TB LZ4 ZSTD Shuffle Read Size (TPCDS 3TB) 0 3 6 9 LZ4 ZSTD 44% Less 43% Less
  26. 26. Q67 Shuffle IO (Cont.) 20% faster QUERY EXECUTION 0 min 5 min 10 min 15 min 20 min LZ4 ZSTD
  27. 27. Apache Parquet ZStandard is smaller than GZIP Storage Apache Parquet (TPCDS 1TB) 0 GB 100 GB 200 GB 300 GB SNAPPY LZ4 GZIP ZSTD
  28. 28. Apache ORC ZStandard is smaller than Parquet ZStandard in general Storage (Cont.) TPCDS 3TB 0 GB 350 GB 700 GB PARQUET ORC
  29. 29. Built-in file format Configurations FORMAT CONFIGURATION PARQUET spark.sql.parquet.compression.codec parquet.compression.codec.zstd.level parquet.compression.codec.zstd.bufferPool.enabled AVRO spark.sql.avro.compression.codec avro.mapred.zstd.level avro.mapred.zstd.bufferpool ORC spark.sql.orc.compression.codec
  30. 30. Agenda ZStandard Issues History When / Why / How to Use Limitations Summary
  31. 31. = This is not a contribution. Limitations ZStandard is not supported by CPU/GPU acceleration Apache ORC is still using ZSTD 1.3.5 - Need to replace aircompressor with zstd-jni Apache Parquet has more room to optimize memory consumption - PARQUET-2022: ZstdDecompressorStream should close `zstdInputStream`
  32. 32. = This is not a contribution. Use ZSTD to maximize your cluster utilizations Summary Use zstd with event log compression by default Use zstd with shuffle io compression with K8s volumes Use zstd with Parquet/ORC/Avro files
  33. 33. TM and © 2021 Apple Inc. All rights reserved.
  34. 34. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.

×