Downloaded 170 times
![© 2017 Dremio Corporation @DremioHQ
Columnar data
persons = [{
name: ’Joe',
age: 18,
phones: [
‘555-111-1111’,
‘555-222-2222’
]
}, {
name: ’Jack',
age: 37,
phones: [ ‘555-333-3333’ ]
}]](https://image.slidesharecdn.com/hadoopsummitsj2017parquetarrowroadmap-170615230306/85/The-columnar-roadmap-Apache-Parquet-and-Apache-Arrow-19-320.jpg){kind=icon}
![© 2017 Dremio Corporation @DremioHQ
Record Batch Construction
Schema
Negotiation
Dictionary
Batch
Record
Batch
Record
Batch
Record
Batch
name (offset)
name (data)
age (data)
phones (list offset)
phones (data)
data header (describes offsets into data)
name (bitmap)
age (bitmap)
phones (bitmap)
phones (offset)
{
name: ’Joe',
age: 18,
phones: [
‘555-111-1111’,
‘555-222-2222’
]
}
Each box (vector) is contiguous memory
The entire record batch is contiguous on wire](https://image.slidesharecdn.com/hadoopsummitsj2017parquetarrowroadmap-170615230306/85/The-columnar-roadmap-Apache-Parquet-and-Apache-Arrow-20-320.jpg){kind=icon}
Uploaded byJulien Le Dem
The columnar roadmap: Apache Parquet and Apache Arrow
This document discusses Apache Parquet and Apache Arrow, open source projects for columnar data formats. Parquet is an on-disk columnar format that optimizes I/O performance through compression and projection pushdown. Arrow is an in-memory columnar format that maximizes CPU efficiency through vectorized processing and SIMD. It aims to serve as a standard in-memory format between systems. The document outlines how Arrow builds on Parquet's success and provides benefits like reduced serialization overhead and ability to share functionality through its ecosystem. It also describes how Parquet and Arrow representations are integrated through techniques like vectorized reading and predicate pushdown.
More Related Content
Similar to The columnar roadmap: Apache Parquet and Apache Arrow
The columnar roadmap: Apache Parquet and Apache Arrow
- 1. © 2017 DremioCorporation @DremioHQ The columnar roadmap: Apache Parquet and Apache Arrow Julien Le Dem, Principal Architect Dremio, VP Apache Parquet, Apache Arrow PMC
- 2. © 2017 DremioCorporation @DremioHQ • Architect at @DremioHQ • Formerly Tech Lead at Twitter on Data Platforms. • Creator of Parquet • Apache member • Apache PMCs: Arrow, Kudu, Incubator, Pig, Parquet Julien Le Dem @J_ Julien
- 3. © 2017 DremioCorporation @DremioHQ Agenda • Community Driven Standard • Benefits of Columnar representation • Vertical integration: Parquet to Arrow • Arrow based communication
- 4. © 2017 DremioCorporation @DremioHQ Community Driven Standard
- 5. © 2017 DremioCorporation @DremioHQ An open source standard • Parquet: Common need for on disk columnar. • Arrow: Common need for in memory columnar. • Arrow is building on the success of Parquet. • Top-level Apache project • Standard from the start: – Members from 13+ major open source projects involved • Benefits: – Share the effort – Create an ecosystem Calcite Cassandra Deeplearning4j Drill Hadoop HBase Ibis Impala Kudu Pandas Parquet Phoenix Spark Storm R
- 6. © 2017 DremioCorporation @DremioHQ Interoperability and Ecosystem Before With Arrow • Each system has its own internal memory format • 70-80% CPU wasted on serialization and deserialization • Functionality duplication and unnecessary conversions • All systems utilize the same memory format • No overhead for cross-system communication • Projects can share functionality (eg: Parquet-to-Arrow reader)
- 7. © 2017 DremioCorporation @DremioHQ Benefits of Columnar representation
- 8. © 2017 DremioCorporation @DremioHQ Columnar layout Logical table representation Row layout Column layout @EmrgencyKittens
- 9. © 2017 DremioCorporation @DremioHQ On Disk and in Memory • Different trade offs – On disk: Storage. • Accessed by multiple queries. • Priority to I/O reduction (but still needs good CPU throughput). • Mostly Streaming access. – In memory: Transient. • Specific to one query execution. • Priority to CPU throughput (but still needs good I/O). • Streaming and Random access.
- 10. © 2017 DremioCorporation @DremioHQ Parquet on disk columnar format
- 11. © 2017 DremioCorporation @DremioHQ Parquet on disk columnar format • Nested data structures • Compact format: – type aware encodings – better compression • Optimized I/O: – Projection push down (column pruning) – Predicate push down (filters based on stats)
- 12. © 2017 DremioCorporation @DremioHQ Parquet nested representation Document DocId Links Name Backward Forward Language Url Code Country Columns: docid links.backward links.forward name.language.code name.language.country name.url Borrowed from the Google Dremel paper https://blog.twitter.com/2013/dremel-made-simple-with-parquet
- 13. © 2017 DremioCorporation @DremioHQ Access only the data you need a b c a1 b1 c1 a2 b2 c2 a3 b3 c3 a4 b4 c4 a5 b5 c5 a b c a1 b1 c1 a2 b2 c2 a3 b3 c3 a4 b4 c4 a5 b5 c5 a b c a1 b1 c1 a2 b2 c2 a3 b3 c3 a4 b4 c4 a5 b5 c5 + = Columnar Statistics Read only the data you need!
- 14. © 2017 DremioCorporation @DremioHQ Parquet file layout
- 15. © 2017 DremioCorporation @DremioHQ Arrow in memory columnar format
- 16. © 2017 DremioCorporation @DremioHQ Arrow goals • Well-documented and cross language compatible • Designed to take advantage of modern CPU • Embeddable – in execution engines, storage layers, etc. • Interoperable
- 17. © 2017 DremioCorporation @DremioHQ Arrow in memory columnar format • Nested Data Structures • Maximize CPU throughput – Pipelining – SIMD – cache locality • Scatter/gather I/O
- 18. © 2017 DremioCorporation @DremioHQ CPU pipeline
- 19. © 2017 DremioCorporation @DremioHQ Columnar data persons = [{ name: ’Joe', age: 18, phones: [ ‘555-111-1111’, ‘555-222-2222’ ] }, { name: ’Jack', age: 37, phones: [ ‘555-333-3333’ ] }]
- 20. © 2017 DremioCorporation @DremioHQ Record Batch Construction Schema Negotiation Dictionary Batch Record Batch Record Batch Record Batch name (offset) name (data) age (data) phones (list offset) phones (data) data header (describes offsets into data) name (bitmap) age (bitmap) phones (bitmap) phones (offset) { name: ’Joe', age: 18, phones: [ ‘555-111-1111’, ‘555-222-2222’ ] } Each box (vector) is contiguous memory The entire record batch is contiguous on wire
- 21. © 2017 DremioCorporation @DremioHQ Vertical integration: Parquet to Arrow
- 22. © 2017 DremioCorporation @DremioHQ Representation comparison for flat schema
- 23. © 2017 DremioCorporation @DremioHQ Naïve conversion
- 24. © 2017 DremioCorporation @DremioHQ Naïve conversion Data dependent branch
- 25. © 2017 DremioCorporation @DremioHQ Peeling away abstraction layers Vectorized read (They have layers)
- 26. © 2017 DremioCorporation @DremioHQ Bit packing case
- 27. © 2017 DremioCorporation @DremioHQ Bit packing case No branch
- 28. © 2017 DremioCorporation @DremioHQ Run length encoding case
- 29. © 2017 DremioCorporation @DremioHQ Run length encoding case Block of defined values
- 30. © 2017 DremioCorporation @DremioHQ Run length encoding case Block of null values
- 31. © 2017 DremioCorporation @DremioHQ Predicate push down
- 32. © 2017 DremioCorporation @DremioHQ Example: filter and projection
- 33. © 2017 DremioCorporation @DremioHQ Naive filter and projection implementation
- 34. © 2017 DremioCorporation @DremioHQ Peeling away abstractions
- 35. © 2017 DremioCorporation @DremioHQ Arrow based communication
- 36. © 2017 DremioCorporation @DremioHQ Universal high performance UDFs SQL engine Python process User defined function SQL Operator 1 SQL Operator 2 reads reads
- 37. © 2017 DremioCorporation @DremioHQ Arrow RPC/REST API • Generic way to retrieve data in Arrow format • Generic way to serve data in Arrow format • Simplify integrations across the ecosystem • Arrow based pipe
- 38. © 2017 DremioCorporation @DremioHQ RPC: arrow based storage interchange The memory representation is sent over the wire. No serialization overhead. Scanner projection/predicate push down Operator Arrow batches Storage Mem Disk SQL execution Scanner Operator Scanner Operator Storage Mem Disk Storage Mem Disk …
- 39. © 2017 DremioCorporation @DremioHQ RPC: arrow based cache The memory representation is sent over the wire. No serialization overhead. projection push down Operator Arrow-based Cache SQL execution Operator Operator …
- 40. © 2017 DremioCorporation @DremioHQ RPC: Single system execution The memory representation is sent over the wire. No serialization overhead. Scanner Scanner Scanner Parquet files projection push down read only a and b Partial Agg Partial Agg Partial Agg Agg Agg Agg Shuffle Arrow batches Result
- 41. © 2017 DremioCorporation @DremioHQ Results - PySpark Integration: 53x speedup (IBM spark work on SPARK-13534) http://s.apache.org/arrowresult1 - Streaming Arrow Performance 7.75GB/s data movement http://s.apache.org/arrowresult2 - Arrow Parquet C++ Integration 4GB/s reads http://s.apache.org/arrowresult3 - Pandas Integration 9.71GB/s http://s.apache.org/arrowresult4
- 42. © 2017 DremioCorporation @DremioHQ Language Bindings Parquet • Target Languages – Java – CPP – Python & Pandas • Engines integration: – Many! Arrow • Target Languages – Java – CPP, Python – R (underway) – C, Ruby, JavaScript • Engines integration: – Drill – Pandas, R – Spark (underway)
- 43. © 2017 DremioCorporation @DremioHQ Arrow Releases 178 195 311 85 237 131 76 17 October 10, 2016 0.1.0 February 18, 2017 0.2.0 May 5, 2017 0.3.0 May 22, 2017 0.4.0 Changes Days
- 44. © 2017 DremioCorporation @DremioHQ Current activity: • Spark Integration (SPARK-13534) • Pages index in Parquet footer (PARQUET-922) • Arrow REST API (ARROW-1077) • Bindings: – C, Ruby (ARROW-631) –JavaScript (ARROW-541)
- 45. © 2017 DremioCorporation @DremioHQ Get Involved • Join the community – dev@{arrow,parquet}.apache.org – Slack: • Arrow: https://apachearrowslackin.herokuapp.com/ • Parquet: https://parquet-slack-invite.herokuapp.com/ – http://{arrow,parquet}.apache.org – Follow @Apache{Parquet,Arrow}