This document discusses how structured data is often moved inefficiently between systems, causing waste. It introduces Apache Arrow, an open standard for in-memory data, and how Arrow can help make data movement more efficient. Systems like Snowflake and BigQuery are now using Arrow to help speed up query result fetching by enabling zero-copy data transfers and sharing file formats between query processing and storage.

2. •
•
•
•
•

3. Some Partners
● https://ursalabs.org
● Apache Arrow-powered
Data Science Tools
● Funded by corporate
partners
● Built in collaboration with
RStudio

4. Systems that move
structured data often
cause significant waste

5. •
•
•

6. •
• …
•

8. Server 1 Server 2 Server 3
Client 1 Client 2

9. Scalable Blob Storage
System 1 System 2 System 3

10. Executor Executor Executor
Executor /
Coordinator
Client
Result Set
Result Set
Result Set
Result Set
Result Set
Result Set

11. Executor Executor Executor Executor
Client
Result Set
Result Set
Result Set

12. •
•
•
•
•

14. •
•
•
•
•

15. •
•
•
•
•

16. •
•
•
•
•
•

17. •
•
•
•

19. •
•
•
•
SCHEMA DICTIONARY DICTIONARY
RECORD
BATCH
RECORD
BATCH

20. •
metadata body

21. https://www.snowflake.com/blog/fetching-query-results-from-snowflake-just-got-a-lot-faster-with-apache-arrow/

22. https://medium.com/google-cloud/announcing-google-cloud-bigquery-version-1-17-0-1fc428512171

23. •
•
•

24. •
•
•
•
•
•

25. •
•
•

26. Client Planner
GetFlightInfo
FlightInfo
DoGet Data Nodes
FlightData
DoGet
FlightData
...

27. •
•
•
message SQLQuery {
binary database_uri = 1;
binary query = 2;
}
Commands.proto GetFlightInfo RPC
type: CMD
cmd: <serialized command>

28. Client
DoGet
Data Node
FlightData
Row
Batch
Row
Batch
Row
Batch
Row
Batch
Row
Batch
...
Data transported in a Protocol
Buffer, but reads can be made
zero-copy by writing a custom
gRPC “deserializer”

29. Mainlining Databases: Supporting Fast Transactional Workloads on
Universal Columnar Data File Formats
Li, Butrovich, Ngom, Lim,
Pavlo, McKinney
https://arxiv.org/pdf/2004.14471.pdf

31. •
•