The document discusses lakehouse architecture, highlighting its evolution from two-tier architecture to a unified data management system that combines the strengths of data lakes and warehouses. It introduces Delta Lake, a managed format enhancing data reliability and performance while outlining its features like ACID transactions and metadata management. Challenges with traditional data formats like Parquet are also addressed, along with Delta Lake's solutions to improve data handling efficiency.

1. Lakehouse
architecture with Delta
Lake & Databricks
Data engineer / Technical & Engineering lead
Dragan Beric

2. Agenda
● Motivation
● Architecture evolution
● Lakehouse
● Delta lake

3. Motivation
Top 3 challenges in enterprise data usage:
○ Access
○ Reliability
○ Timeliness

4. Architecture evolution
two-tier architecture

5. Architecture evolution
two-tier architecture
• flexible storage
• low - cost storage
• easy access for
ML & DS

6. Architecture evolution
two-tier architecture
• flexible storage
• low - cost storage
• easy access for
ML & DS
• data duplication
• additional jobs for
data movement
• maintenance
(data +
development)

7. Architecture evolution
lakehouse architecture

8. Architecture evolution
lakehouse architecture
• flexible storage
• low - cost storage
• easy access for ML
& DS
• single source of truth
• governance
• schema validation
• ACID transactions
over files

9. Lakehouse
A lakehouse is a new, open architecture that
combines the best elements of data lakes and data
warehouses

10. Lakehouse
Lorem ipsu
datawarehouse
Fast
Structured
Operational
Governed
Cheap
Flexible
Scalable
data lake
Lakehouse

11. Delta lake
what is delta?
Databricks delta is a unified data
management system that brings
reliability and performance to
existing data lakes

12. Delta lake
what is delta?
Databricks delta is a unified data
management system that brings
reliability and performance to
existing data lakes
Delta Lake is an optimized,
managed format for organizing &
working with Parquet files

13. Delta lake
what is delta?
Databricks delta is a unified data
management system that brings
reliability and performance to
existing data lakes
Delta Lake is an optimized,
managed format for organizing &
working with Parquet files
“It’s Parquet, just better!”

14. Delta lake
challenges with parquet
• Hard to append data
• Update / Merge not supported
• Metadata does not scale (a lot
of small files)
• A lot of small parquet files (no
auto-compaction)
• Jobs failing mid way

15. Delta lake
structure
folder that contains:
• data in parquet
• delta log
• data

16. Delta lake
structure
folder that contains:
• data in parquet
• delta log
• data

17. Delta lake
delta log – transactional layer
folder that contains:
• table-schema
• commits info
• meta-data
checkpoint file
10 commits
(transactions)

18. Delta lake
query execution plan
Query is
received
Processing
_delta_log
Find and read
latest
checkpoint file
Read
transactions
after the
checkpoint
Read data
referenced by
checkpoint and
transactions
Return results

19. Delta lake
DML operations – merge/update
_delta_log
storage
delta
lake
Product Price($)
Apple 1.5
Banana 0.53
Lemon 1.42
UPDATE DimProduct
SET Price = 1$
WHERE Product = ‘Lemon’
0000.json
“add”:{“part-01.parquet”,…}
part-01
3 rows
0001.json
“remove”:{“part-01.parquet”,…},
“add”:{“part-02.parquet”,…}
part-01
3 rows
part-02
3 rows
Product Price($)
Apple 1.5
Banana 0.53
Lemon 1

20. Delta lake
DML operations – delete
_delta_log
storage
delta
lake
Product Price($)
Apple 1.5
Banana 0.53
Lemon 1.42
DELETE from DimProduct
WHERE Product = ‘Lemon’
part-01
3 rows
0002.json
“remove”:{“part-02.parquet”,…},
“add”:{“part-03.parquet”,…}
part-01
3 rows
part-02
3 rows
Product Price($)
Apple 1.5
Banana 0.53
0001.json
“remove”:{“part-01.parquet”,…},
“add”:{“part-02.parquet”,…}
part-02
3 rows
part-03
2 rows

21. Delta lake
DML operations – insert
_delta_log
storage
delta
lake
INSERT INTO DimProduct
VALUES (‘Mango’,3)
0003.json
“add”:{“part-04.parquet”,…}
part-01
3 rows
part-02
3 rows
Product Price($)
Apple 1.5
Banana 0.53
Mango 3
part-03
2 rows
Product Price($)
Apple 1.5
Banana 0.53
0002.json
“remove”:{“part-02.parquet”,…},
“add”:{“part-03.parquet”,…}
part-01
3 rows
part-02
3 rows
part-03
2 rows
part-04
1 rows

22. Delta lake
features – conclussion
ACID
transactions
indexing
metadata
scaling
governance
schema
validation
time travel

23. HTEC group | USA, San Francisco | 535 Mission St, 14th floor | +1 415 490 8175 | office-sf@htecgroup.com | htecgroup.com
Q&A