Hive tables are an integral part of the big data ecosystem, but the simple directory-based design that made them ubiquitous is increasingly problematic. Netflix uses tables backed by S3 that, like other object stores, don’t fit this directory-based model: listings are much slower, renames are not atomic, and results are eventually consistent. Even tables in HDFS are problematic at scale, and reliable query behavior requires readers to acquire locks and wait. Owen O’Malley and Ryan Blue offer an overview of Iceberg, a new open source project that defines a new table layout addresses the challenges of current Hive tables, with properties specifically designed for cloud object stores, such as S3. Iceberg is an Apache-licensed open source project. It specifies the portable table format and standardizes many important features, including: * All reads use snapshot isolation without locking. * No directory listings are required for query planning. * Files can be added, removed, or replaced atomically. * Full schema evolution supports changes in the table over time. * Partitioning evolution enables changes to the physical layout without breaking existing queries. * Data files are stored as Avro, ORC, or Parquet. * Support for Spark, Hive, and Presto.

2. ●
●
○
○
●
○
○
●

4. ●
○
○
○
●
select distinct tags['type'] as type
from iceberg.atlas
where
name = 'metric-name' and
date > 20180222 and date <= 20180228
order by type;

5. ●
○
○
●
○
○
●
○
○

8. ●
○
●
○
○
○
○

9. ●
●
○
○
○
●
○
○

11. ●
○
date=20180513/
|- hour=18/
| |- ...
|- hour=19/
| |- 000000_0
| |- ...
| |- 000031_0
|- hour=20/
| |- ...
|- ...

12. ●
SELECT ... WHERE date = '20180513' AND hour = 19
date=20180513/
|- hour=18/
| |- ...
|- hour=19/
| |- 000000_0
| |- ...
| |- 000031_0
|- hour=20/
| |- ...
|- ...

13. ●
○
○
○
●
○
●
○
●
○

14. ●
●
●
○
date=20180513/
|- hour=19/
| |- base_0000000/
| | |- bucket_00000
| | |- ...
| | |- bucket_00031
| |- delta_0000001_0000100/
| | |- bucket_00000
| | |- ...

15. ●
○
○
●
●
●
●
○
○

16. ●
○
○ null __HIVE_DEFAULT_PARTITION__
●
○
●
●

17. ●
○ ts > X ⇒
○
ts > X and (d > day(X) or (d = day(X) and hr >= hour(X))
●
○
●
○
○

19. ●
○
○
S1 S2 S3 ...

20. ●
○
○
●
○
○
S1 S2 S3 ...
R W

22. ●
○
○
●
●
●
v1.json
S1 S2
v2.json
S1 S2 S3
v3.json
S2 S3

23. ●
○
○
○
v2.json
S1 S2 S3
m0.avro m1.avro m2.avro

24. ●
○
○
●
○
○
●
○
○

25. ●
○
○
○
●
●
○
○

26. ●
○
○
●
○
○
●

27. ●
○ file1.avro file2.avro
○ merge1.parquet
●
○ file1.avro file2.avro
○
file1.avro file2.avro
merge1.parquet
● file1.avro file2.avro

28. ●
●
○
○
●
○ O(1) O(n)
○
○

29. ●
●
○
○
●
○
○
●

30. ●
○
○
●
○
○
●
○
○

32. ●
○
○
○
●
○
○
●
○

33. ●
○
●
●