Apache Parquet is a columnar storage format that can efficiently store nested data. It allows nested fields to be read independently of other fields. Many data processing systems like Hive, Spark, Pig and MapReduce can understand the Avro format used by Parquet. Parquet uses encoding and compression techniques like run-length encoding, dictionary encoding, and compression algorithms like Snappy and gzip to improve file size and query performance. The Parquet file format consists of a header, blocks with columns and pages, and a footer to store metadata.

1. Apache Parquet
https://parquet.apache.org/

2. Why Parquet
Columnar storage format
Can store nested Data
Efficiency in file size and query performance
Nested field can be read independently of other fields
Many data processing understand avro format (Hive, Spark, Pig and
MapReduce,etc)

3. Data Model
boolean int32 int64 int96 float double binary fixed_len
_byte_ar
ray
UTF8 ENUM DECIMAL(pr
ecision,scale
DATE LIST MAP

4. Record parquet
message Nom {
(required,optional,repeated) type nom ;
required int32 age ;
required binary nom (UTF8)
}
{"name": "right", "type":
"string", "order":
"descending"}
Avro
Parquet

5. Parquet File Format
Header Block .. Block Footer
Magic number
Using parquet
Magic number
,Schema,Encoding
method,Block position,..
Columns
Pages
128MB

6. Encoding & Compression
Run-length : True , True ,True ,False (3 true , 1 false)
Dictionary encoding ( indexation )
Plain encoding
Compression algorithms Snappy,gzip,...

7. Configuration
parquet.block.size int
parquet.page.size int
parquet.dictionary.page.size int
parquet.enable.dictionary boolean true
parquet.compression String (SNAPPY,UNCOMPRESSED,LZO...)

8. Writing and reading Parquet files

9. Parquet MapReduce

10. Parquet MapReduce

11. Parquet MapReduce