Apache Arrow is an open source project intended to provide a standardized columnar memory format for flat and hierarchical data. It enables more efficient analytics workloads for modern CPU and GPU hardware, which makes working with large data sets easier and cheaper. InfluxData and Dremio are both members of the Apache Software Foundation (ASF). Dremio is a data lakehouse management service known for its scalability and capacity for direct querying across diverse data sources. InfluxDB is the purpose-built time series database, and InfluxDB 3.0 has a new columnar storage engine and uses the Arrow format for representing data and moving data to and from Parquet. Discover how InfluxDB and Dremio have advanced their solutions by relying on the Apache Arrow framework. Join this live panel as Alex Merced and Anais Dotis-Georgiou dive into: Advantages to utilizing the Apache Arrow ecosystem Tips and tricks for implementing the columnar data structure How developers can best utilize the ASF to innovate and contribute to new industry standards

1. | © Copyright 2023, InfluxData
How InfluxDB and
Dremio Leverage the
Apache Ecosystem
July 2023

2. | © Copyright 2023, InfluxData
2
Anais Dotis-Georgiou
Developer Advocate
LinkedIn

3. | © Copyright 2023, InfluxData
3
Alex Merced
Developer Advocate
LinkedIn

4. | © Copyright 2023, InfluxData
InfluxDB & Apache
4

5. | © Copyright 2023, InfluxData
InfluxData Reference Architecture

6. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
InfluxDB’s new storage engine is built on
● Rust
● Apache Arrow
● Apache Parquet
● Arrow Flight
● DataFusion

7. | © Copyright 2023, InfluxData
Arrow and InfluxDB
7

8. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
Arrow and InfluxDB
● Overcomes memory challenges. Fine grain memory control.
● Efficient data exchange provides for data analytics
performance benefits
● Arrow provides the in-memory columnar storage while
Parquet will provide the column-oriented data file format on
disk.
● Broad ecosystem compatibility
8

9. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
Sidebar–Advantages of Columnar Data Storage
9
measurement1,tag1=tagvalue1 field1=1i timestamp1
measurement1,tag1=tagvalue2 field1=2i timestamp2
measurement1,tag2=tagvalue3 field1=3i timestamp3
measurement1,tag1=tagvalue1,tag2=tagvalue3 field1=4i,field2=true timestamp4
measurement1, field1=1i timestamp5

10. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
Sidebar–Advantages of Columnar Data Storage
10
Name: measurement1
field1 field2 tag1 tag2 tag3 time
1i null tagvalue1 null null timestamp1
2i null tagvalue2 null null timestamp2
3i null null tagvalue3 null timestamp3
4i true tagvalue1 tagvalue3 tagvalue4 timestamp4
1i null null null null timestamp5

11. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
Sidebar–Advantages of Columnar Data Storage
11
1i 2i 3i 4i 1i
null null null true null
tagvalue1 tagvalue2 null tagvalue1 null
null null tagvalue3 tagvalue3 null
null null null tagvalue4 null
timestamp1 timestamp2 timestamp3 timestamp4 timestamp5
1i, 2i, 3i, 4i, 1i;
null, null, null, true, null;
tagvalue1, tagvalue2, null, tagvalue1, null;
null, null, null, tagvalue3, tagvalue3, null;
null, null, null, tagvalue4, null;
timestamp1, timestamp2, timestamp3, timestamp4, timestamp5.

12. | © Copyright 2023, InfluxData
DataFusion and InfluxDB
12

13. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
DataFusion and InfluxDB
● DataFusion provides the query, processing, and
transformation of the data.
● Enables fast query against data stored on cheaper object
store.
● DataFusion supports both a postgres compatible SQL and
DataFrame API.
13

14. | © Copyright 2023, InfluxData
Parquet and InfluxDB
14

15. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
Parquet and InfluxDB
● Efficient compression and interoperability with ML and
analytics tooling.
● Take up little disk space and are fast to scan.
● Enable bulk data export and import.
● Offer interoperability with modern ML and analytics tools.
15

16. | © Copyright 2023, InfluxData
Dremio & Apache
16

17. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
What is Dremio
17
Test-Drive Dremio

18. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
Dremio and
18
Apache Calcite is an open-source framework for building databases and data management systems. It allows developers to
design systems that can parse, optimize, and execute SQL queries. Apache Calcite provides components to implement
SQL-like query processing capabilities in various systems.
- SQL Parser
- Query Optimizer
- Adapter Framework
- Pluggable Query Execution
- Standards Support
Enables Dremio’s
SQL Interface

19. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
Dremio and
19
- Standard In-memory format
- Standard Transport Protocol
(Arrow Flight)
- Pre-Compiled Query Logic on
Arrow Buffers (Gandiva)
Enables Dremio’s
Data Processing

20. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
Dremio and Apache
20
- Metadata Layer for Data Lake
Datasets
- Enables Time Travel
- Enables Schema Evolution
- Enables Partition Evolution
Enables Dremio Data
Reflections and More

21. | © Copyright 2023, InfluxData
| © Copyright 2023, InfluxData
Dremio and Apache
21
- Binary
- Columnar
- Open Standard
- Predicate Pushdown
Enables Dremio’s
Caching Layer

22. | © Copyright 2023, InfluxData
Get Help + Resources!
22
Forums: community.influxdata.com
Slack: influxcommunity.slack.com
GH: github.com/InfluxCommunity
Book: awesome.influxdata.com
Docs: docs.influxdata.com
Blogs: influxdata.com/blog
InfluxDB University: influxdata.com/university

23. | © Copyright 2023, InfluxData
23
R E S O U R C E S
Webinar: InfluxDB 3.0 Basics
Learn the basics of InfluxDB, covering core time series
concepts, an overview of the platform, and practical
guidance for installation and usage.
Save 96% on Data Storage Costs:
bit.ly/3NJEcGZ
Run a Proof of Concept:
InfluxDB Resources
Watch now
Learn more
Learn more bit.ly/3puRsal
bit.ly/445mRy4

24. | © Copyright 2023, InfluxData
Questions?
24

25. | © Copyright 2023, InfluxData
T H A N K Y O U