Dali: Data Access Layer talk from May 2018 Hive User Group Meeting @Cloudera - https://www.meetup.com/Hive-User-Group-Meeting/events/249641278/

1. Dali: Data Access Layer
​Adwait Tumbde
​LinkedIn

2. Outline
• Complexity of data ecosystem
• Dali to tame the complexity
• Architecture
• How we leverage Hive
• Future

3. Data @ LinkedIn
3
Multiple Formats
(Avro, ORC)
Reuse Logic
across engines
Offline, Nearline &
OnlineDB HDFS
Pig
Map
Reduce
Spark … Samza
REST Stream Files

4. Dali: Data Access Layer
4
DB HDFS
Pig
Map
Reduce
Spark … Samza
REST Stream Files
Dali Datasets
Data
Model
Registry
Readers
Tooling

5. Tables, Views, and Dali Views
5
Table
Path
Format
Schema
Partitioning Scheme
…
View f(table(s)|view(s))
Dali
View
f(table(s)|view(s))
UDFs
Dependencies

6. Sample Dali Dataset
CREATE VIEW profile_flattened
​TBLPROPERTIES(
​ 'functions' =
​ 'get_profile_section:isb.GetProfileSections',
​ 'dependencies' =
'com.linkedin.dali-udfs:get-profile-sections:0.0.5'
)
​AS SELECT get_profile_section(...)
​FROM prod_identity.profile;

7. Dali Reader API Architecture
7
select
f(requestheader,
“/dim_page_key”)
from (
select * from
tracking.webAds
union all
select * from
tracking.mobileAds
) t where
t.requestheader.pageke
y IN (“pk1”, “pk2” …)
Hive
To Target Type
Dali Table Reader
Target Platform Execution
HDFS
Union
Sel
Filter
Sel
Filter
Sel
Dali View Engine

8. Dali – Complete Picture
CREATE VIEW v
AS SELECT …
Hive
Metastore
Artifactory
Tooling Deployer
Pig
Map
Reduce
Spark … Samza
Dali Datasets
REST Stream Files

9. Why Dali?
9
Write once
Run Anywhere
Compliance
Public – Private
interfaces for Data

10. Limitations
• Restricted to map-side operations
• No joins or aggregations … for now
10

11. 11
Future: Standard SQL and Portable UDFs
Calcite
Relational
Algebra
Portable
UDFs
Catalog

12. Dali Team
• Carl Steinbach
• Anthony Hsu
• Walaa Eldin Moustafa
• Shardul Mahadik
• Ratandeep Ratti
• Rohan Agarwal
• Chris Chen
• Vasanth Rajamani

13. Questions
• Email:
atumbde@linkedin.com
• Blog
Dali Views: Functions as a Service for Big Data
13

14. ©2014 LinkedIn Corporation. All Rights Reserved.©2014 LinkedIn Corporation. All Rights Reserved.