Presented by Eric Ogren and Anthony Hsu at the LinkedIn Big Data Meetup on 2018-01-25: https://www.meetup.com/Big-Data-Meetup-LinkedIn/events/246858500/

1. Balancing Data Democracy with Data
Privacy: The LinkedIn Story
Jan 25, 2018
Eric Ogren
Anthony Hsu
Big Data Meetup, LinkedIn SF
1

2. We needed data democracy to
deliver member value
LinkedIn Data Science
I want to analyze as much data as
possible so my models are accurate
Data Democracy
ALL THE DATA, ALL THE TIME
I want to discover data that’s needed for my
analysis as fast as possible
I want to access that data as quickly as
possible for my analysis
2

3. I want my personal data to be stored only
where needed and not propagated
unnecessarily
Data Protection
Need to Ensure Member Privacy
LinkedIn Members
STORE, PROCESS, DELETE,..
I want my personal data to be deleted when
I close my account or request deletion
I want my personal data to only be
processed if essential and only if I consent
3

4. DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The Data Paradox
4

5. DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The Data Paradox
5

6. Data Hubs at LinkedIn
In Motion
At Rest
Scale
O(10) clusters
~2.3 Trillion messages / day
~450 TB written / day
Scale
O(10) clusters
~10K machines
~XXX PB at rest
6

7. In Motion
At Rest
Data Integration
SFTP
JDBC
REST
Azure
Blob, Data
Lake
Storage
7

8. REQUIREMENTS
Less Data
Legal: Right to Erasure or Right to be Forgotten
“Delete all my personal data without undue delay when it is no
longer necessary / when consent has been withdrawn”
Engineering:
Need the ability to delete some specific subset or all data associated
with a specific LinkedIn member from all our data systems
8

9. A lot of data, different formats
Challenges
Understand HDFS data: organization, formats, …
Cycle asynchronously, within an SLA, deleting
records, without affecting running jobs
Quarantine exceptional records for manual triage
Can scale to processing hundreds of PB of data
Data Deletion
IMPLICATIONS FOR HADOOP
9

10. Gobblin: The Logical Pipeline
Source
Work
Unit
Work
Unit
Work
Unit
Extract Convert Quality Write Data
Publish
WriteQualityConvertExtract
Extract Convert Quality Write
Task
Task
Task
10

11. Gobblin: Extending for Purge
HDFS
Work
Unit
Data
Publish
Extract Convert Quality Write
Task
Task
HDFS
If needs purge
then drop
else continue
Member’s Delete
Requests
11

12. STATUS AND CHALLENGES
Gobblin: Data Lifecycle Management at Scale
Status
Number of datasets: many thousands
Amount of data scanned for purge: hundreds of TB/day
Challenges
Immutable Storage Formats + Right to Erasure = Unhappy Disks
“Widespread implementation will surely lead to innovation in these formats!”
12

13. DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The Data Paradox
DATA LIFECYCLE MANAGEMENT
13

14. DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The Data Paradox
DATA LIFECYCLE MANAGEMENT
14

15. LinkedIn’s Data Ecosystem
15

16. Metadata based Search Experience
for Data Scientists
Data Discovery
Where is dataset X?
How did it get created?
Usage : In production since 2014
Users : Data Scientists, Product Engineers
Use Cases: Discovery, Impact Analysis
WhereHows
FIND DATA, NAVIGATE RELATIONSHIPS
Open source @ github.com/linkedin/wherehows 16

17. SEARCH SCREENSHOTS
WhereHows
17

18. LINEAGE SCREENSHOTS
WhereHows
18

19. More than just Discovery
Use Cases
Which datasets at LinkedIn contain PII or highly
confidential data?
How many contain member-member messages?
How many of them are accessible by team X?
Have all datasets been purged within SLA?
Discovering Violations
ANSWERING HARDER QUESTIONS
19

20. Wide + Deep
Metadata
Comprehensive coverage of data systems at LinkedIn
We have > 20 systems!
SQL, NoSQL, Indexes, Blob Stores, …
Deeper understanding of each dataset
Schema is not enough
Need to understand semantics
Discovering Violations
REQUIREMENTS
20

21. A METADATA REFINERY APPROACH
WhereHows Architecture @ 10,000 ft
ML driven
refinements
21

22. METADATA SHOULD LOOK JUST LIKE DATA
WhereHows Architecture @ 10,000 ft
ML driven
refinements
Unified Metadata Dataset
Metadata Serving Repository
key-value
search
graph
Data Systems
Technical metadata
Snapshots
Stream
Services + Jobs
Operational
Metadata
WhereHows
Application
LinkedIn
Community AnnotationsTechnical metadata
Data Catalogs
Process Definitions
Code
Operational metadata
Data Publish
Data Access
Job Executions
22

23. DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The Data Paradox
DATA LIFECYCLE MANAGEMENT
METADATA
23

24. METADATA
DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The Data Paradox
DATA LIFECYCLE MANAGEMENT
24

25. Simple to Complex
Different Types
Basic Restrictions
Access to dataset based on business need
Privacy by Default
Analysts shouldn’t get access to raw PII
(Personally Identifiable Information) by default
Consent-based Access
Access to certain data elements only available
if member has consented for that particular use-
case
Access Restrictions
REQUIREMENTS
25

26. FREEDOM OF EXPRESSION
Many Transformation Engines @ LinkedIn
In Motion
At Rest
26

27. HARD TO CHANGE ANYTHING UNDERNEATH!
Challenge for Infrastructure Providers
(Pig scripts)
My Raw Data
Native readers, dependencies on path, format hard-coded
Hard to move to
better formats
without breaking
everyone or
copying data twice
My Raw Data
27

28. HARD TO CHANGE ANYTHING UPSTREAM!
Semantic Challenges
Data is unclean (bad data on certain dates)
Data models are in constant flux (split event into multiple)
Have to change
data processing
logic everywhere!
My Raw Data
28

29. AN API TO MANAGE EVOLUTION
We need “microservices” for Data
My Data API
My Raw Data
29

30. A DATA ACCESS LAYER FOR LINKEDIN
We built Dali to solve this
Dataset Readers
Dataset Tooling
Abstract away underlying physical details to
allow users to focus solely on the logical
concerns
30

31. Dali: Implementation Details in Context
Dataflow APIs
(MR, Spark,
Scalding)
Query Layers
(Pig, Hive,
Spark)
Data CatalogGit + Artifactory
Dataset
Owner
31
Datasets
+
UDFs
Dali Datasets (Tables+Views)
Dali Readers

32. STEP 1: DATA + METADATA
Solving for Compliant Access
Schema = {
int memberId
String firstName
String lastName
Position[] positions
educationHistory[] educationHistory
…
}
MemberProfile
NAME : is_pii
MEMBER_ID : is_pii
Raw
Dataset
Meta
Data
32

33. STEP 2: A MEMBER’S PREFERENCES
Privacy Preferences
33

34. A BITMAP DATASET: ONE PER MEMBER PER SETTING
Privacy Preferences
34

35. A BITMAP DATASET: ONE PER MEMBER PER SETTING
Privacy Preferences
35
Member Privacy
Preferences

36. Solving for Compliant Access With Dali
Raw
Dataset
Meta
Data
Member Privacy
Preferences
Dali Reader responsibility:
Given:
(Dataset, Metadata, UseCase)
Generate:
Dataset and Column-level
transformations
(obfuscate, null, …)
Auto-join with Member
Privacy Preferences
(filter out data elements that
are not consented to)
Processing
Logic
Dali
Reader
Library
Use
Case = X
36

37. Compliance Transformations: Under the Hood
37
Table Scan Operator
Filter Operator
Select Operator
Table Scan Operator
Filter Operator
Select Operator
GDPR Operator
Meta
Data
Query
Context
Privacy
Settings

38. Solving for Compliant Purging With Dali + Gobblin
Raw
Dataset
Meta
Data
Member Privacy
Preferences
Gobblin
Purger
Dali
Reader
Library
Use
Case =
Purge
Purged
Dataset
Member Delete
Requests
38

39. DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The Data Paradox
DATA LIFECYCLE MANAGEMENT
METADATA
DATA ACCESS LAYER
39

40. DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The Data Paradox : Solved !
METADATA
DATA ACCESS LAYER
DATA LIFECYCLE MANAGEMENT
40

41. DATA DEMOCRACY + DATA PROTECTION
The Technology Blueprint
WhereHows*
Dali Apache Gobblin*
* Open Source : We can collaborate on these together!
DATA LIFECYCLE MANAGEMENTDATA ACCESS LAYER
METADATA
41

42. Thank You!
42