The document discusses Merck's data governance initiative to implement Hadoop and establish data governance policies and processes. It outlines their 90 day and 6 month roadmaps to set up a data lake, implement data cataloging and metadata management, and establish access controls and data lineage tracking to enable self-service analytics while ensuring data governance and security. Key components include Hadoop, Hive, HDFS, Ranger, Falcon, and metadata stored in the Hive metastore.

1. Hadoop in Validated Environment
Data Governance Initiative

2. Martin Ryzl
Director, Analytics Platform
Ivo Lasek
Architect, Analytics Platform

3. Research
Manufacturing
Marketing

4. Search
Data
Integration
Data
Analytics
Open
Data

5. 90 Days

6. Laboratory
Information
Management
SAP
Enterprise
Asset
Management
Manufacturing
Execution
Systems
Data
Analytics
Data
Integration

7. Who is the
dataset owner?
How can I
get access?
What does the
data mean?
How can
I reproduce
the results?
Where is the
data I need?

8. 6 Months

9. Where Is the Data I Need?

10. Data Lake

11. Data Lake
Merge
Clean

12. Data Lake
Merge
Clean
Security and Data Governance

13. Data Catalog
Source: http://www.data.gov/

14. Who Is the Dataset Owner?

15. Entitlements
Dataset
Owner
Dataset
User

16. Entitlements
Dataset
Owner
Dataset
User

17. Entitlements
Dataset
Owner
Dataset
User

18. How Can I Get Access?

19. Entitlements
Dataset
Owner
Entitlement
Steward

20. Entitlements
Dataset
Owner
Entitlement
Steward
Dataset
User

21. Entitlements
Dataset
Owner
Entitlement
Steward
Dataset
User

22. Entitlements
Dataset
Owner
Entitlement
Steward
Dataset
User

23. What Does the Data Mean?

24. Semantic meaning – Metastore
id name ssn birth_n
o
phone id
personal_number
employee_number
first_name
division

25. Metastore
Dataset
Owner

26. Metastore
Dataset
Owner
Data
Steward

27. Metastore
Dataset
Owner
Data
Steward

28. Metastore
Dataset
Owner
Data
Steward
Entitlement
Steward
Dataset
User

29. Metastore
Dataset
Owner
Data
Steward
Entitlement
Steward
Dataset
User

30. Metastore
Dataset
Owner
Data
Steward
Entitlement
Steward
Dataset
User

31. How Can I Reproduce the Data?

32. Reporting Data
Reproducibility
delta1
delta2
delta3
Raw Data
Aggregate
v1.0
delta1..delta3
Reporting Data

33. Reproducibility
delta1
delta2
delta3
Raw Data
delta1..delta3
Reporting Data
Aggregate
v1.0
delta4
delta5
delta6 delta1..delta7
Reporting Data
Aggregate
v1.0
delta7

34. Traceability
delta1
delta2
delta3
Raw Data
delta1..delta3
Reporting Data
Aggregate
v1.0
delta1..delta3
Cleaned Data
Clean
v1.0
delta4
delta5
delta6 delta1..delta7
Reporting Data
Aggregate
v1.0
delta1..delta7
Cleaned Data
Clean
v1.1
delta7

35. Data Lineage

36. Access Logs

37. Process Logs
2015-06-04 12:53:31,601 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1702)) - Get metadata for subqueries
17865102 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Get metadata for destination tables
2015-06-04 12:53:31,601 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1726)) - Get metadata for destination
tables
17865345 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Completed getting MetaData in Semantic Analysis
2015-06-04 12:53:31,844 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(10004)) - Completed getting MetaData
in Semantic Analysis
17865347 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Not invoking CBO because the statement has too few joins
2015-06-04 12:53:31,846 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:canHandleAstForCbo(10258)) - Not invoking CBO
because the statement has too few joins
Heart beat
17866695 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException [Error 10044]: Line 2:18 Cannot insert into target
table because column number/types are different ''2015-06-04-07-50'': Table insclause-0 has 165 columns, but query has 166 columns.
org.apache.hadoop.hive.ql.parse.SemanticException: Line 2:18 Cannot insert into target table because column number/types are different
''2015-06-04-07-50'': Table insclause-0 has 165 columns, but query has 166 columns.
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genConversionSelectOperator(SemanticAnalyzer.java:6535)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6336)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8977)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8868)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9713)

38. Who is the
dataset owner?
How can I
get access?
What does the
data mean?
How can
I reproduce
the results?
Where is the
data I need?
HDFS/Hive Metastore Ranger
Metastore Falcon

39. Contacts
• Martin Ryzl (martin.ryzl@merck.com)
• Ivo Lasek (ivo.lasek@merck.com)
• http://www.merck.com/
• http://www.msdit.cz/