The document outlines Datadog's use of Parquet for efficiently collecting and processing vast amounts of metrics data from cloud applications. It highlights key aspects such as their data pipeline, the benefits of using Parquet, and some production insights related to storage efficiency and read performance. The discussion emphasizes the importance of separate compute and storage as well as a standard data format for optimal performance.

1. Parquet at Datadog
How we use Parquet for tons of metrics data
Doug Daniels, Director of Engineering

2. Outline
• Monitor everything
• Our data / why we chose Parquet
• A bit about Parquet
• Our pipeline
• What we see in production

3. Datadog is a monitoring
service for large scale cloud
applications

4. Collect Everything
Integrations for 100+ components

5. Monitor Everything

6. Alert on Critical Issues Collaborate to Fix them Together
Monitor Everything

7. We collect a lot of data

8. We collect a lot of data…
the biggest and most
important of which is

9. Metric timeseries data
timestamp 1447020511
metric system.cpu.idle
value 98.16687

10. We collect
hundreds of billions
of these per day
…and growing every week

11. And we do massive
computation on them

12. • Statistical analysis
• Machine learning
• Ad-hoc queries
• Reporting and aggregation
• Metering and billing

13. One size does not fit all.

14. ETL and aggregation Pig / Hive
ML and iterative algorithms Spark
Interactive SQL Presto
We want the best framework
for each job

15. How do we do that?
Duplicating data storage
Writing redundant glue code
Copying data definitions and schema

16. 1. Separate Compute and Storage
• Amazon S3 as data system-of-record
• Ephemeral, job-specific clusters
• Write storage once, read everywhere

17. 2. Standard Data Format
• Supported by major frameworks
• Schema-aware
• Fast to read
• Strong community

19. Parquet is a column-oriented
data storage format

20. What we love about Parquet
• Interoperable!
• Stores our data super efficiently
• Proven at scale on S3
• Strong community

21. Quick Parquet primer
Column A
Row Group 0
Page 0
Page 1
Page 2
Column B
Page 0
Page 1
File Meta Data
Footer
Row Group 0 Metadata
Column B Metadata
…
Column A Metadata

22. Efficient storage and fast reads
• Space efficiencies (per page)
• Type-specific encodings: run-length, delta, …
• Compression
• Query efficiencies (support varies by framework)
• Projection pushdown (skip columns)
• Predicate pushdown (skip row groups)
• Vectorized read (many rows at a time)

23. Broad ecosystem support

24. Our Parquet pipeline
Kafka
- Buffer
- Sort
- Dedupe
- Upload
Go
Hadoop Spark Presto
PrestoS3FileSystemEMRFS
- Partition
- Write Parquet
- Update Metastore
Luigi/Pig
Metadata
Hive Metastore
csv-gz
Amazon S3
Parquet

25. What we see in production

26. Excellent storage efficiency
• For just 5 columns:
• 3.5X less storage than gz-compressed CSV
• 2.5X less than internal query-optimized columnar format

27. …a little too efficient
• One 80MB parquet file with 160M rows / row group
• Creates long-running map tasks
• Added PARQUET-344 to limit rows per row group
• Want to switch this to limit by uncompressed size

28. Slower read performance
with AvroParquet
Runtime for our test job (mins)
0 min
10 min
20 min
30 min
40 min
C
SV
+
gz
AvroParquet+
gz
AvroParquet+
snappy
Parquet+
gz
• Tried reading schema w/
AvroReader
• Saw 3x slower reads with
AvroParquet (YMMV) on jobs
• Using HCatalog reader + hive
metastore for schema in
production

29. Our Parquet configuration
• Parquet block size (and dfs block size): 128 MB
• Page size: 1 MB
• Compression: gzip
• Schema Metadata: pig (we actually use hive metastore)

30. Thanks!
Want to work with us on Spark, Hadoop,
Kafka, Parquet, Presto, and more?
DM me @ddaniels888 or doug@datadoghq.com