From Hadoop Summit 2015, San Jose From Apache BigData 2016, Vancouver Hadoop has long had strong authentication via integration with Kerberos, authorization via User/Group/Other HDFS permissions, and auditing via the audit log. Recent developments in Hadoop have added HDFS file access control lists, pluggable encryption key provider APIs, HDFS snapshots, and HDFS encryption zones. These features combine to give important new data protection features that every company should be using to protect their data. This talk will cover what the new features are and when and how to use them in enterprise production environments. Upcoming features including columnar encryption in the ORC columnar format will also be covered.

1. © Hortonworks Inc. 2016
Protecting Enterprise Data
in Apache Hadoop
May 2016
Page 1
Owen O’Malley
owen@hortonworks.com
@owen_omalley

2. © Hortonworks Inc. 2016
Security
Page 2

3. © Hortonworks Inc. 2016
Threat: Accidental File Deletion
Page 3

4. © Hortonworks Inc. 2016
Threat: Accidental Killing Tasks
Page 4

5. © Hortonworks Inc. 2016
Threat: Pretending to be a User
Page 5

6. © Hortonworks Inc. 2016
Threat: User accesses private data
Page 6

7. © Hortonworks Inc. 2016
Threat: Pretending to be a Service
Page 7

8. © Hortonworks Inc. 2016
Threat: Remote Access
Page 8

9. © Hortonworks Inc. 2016
Security Architecture
Page 9

10. © Hortonworks Inc. 2016
Threat: Eavesdropping Inside Data Center
Page 10

11. © Hortonworks Inc. 2016
Threat: Eavesdropping Outside Data Center
Page 11

12. © Hortonworks Inc. 2016
Threat: Physical access
Page 12

13. © Hortonworks Inc. 2016
Threat: Bad Hadoop Admin in Cluster
Page 13

14. © Hortonworks Inc. 2016
HDFS Encryption
Page 14

15. © Hortonworks Inc. 2016
KeyProvider API
Page 15

16. © Hortonworks Inc. 2016
Encryption Scheme
Page 16

17. © Hortonworks Inc. 2016
Original Hive Architecture
Page 17

18. © Hortonworks Inc. 2016
Threat: User Accesses DB directly
Page 18

19. © Hortonworks Inc. 2016
Hive Architecture with Metastore
Page 19

20. © Hortonworks Inc. 2016
Threat: User Deletes Hive tables
Page 20

21. © Hortonworks Inc. 2016
Hive Architecture with Storage-Based Auth
Page 21

22. © Hortonworks Inc. 2016
Threat: User reads private columns
Page 22

23. © Hortonworks Inc. 2016
Hive Architecture with Hive Server 2
Page 23

24. © Hortonworks Inc. 2016
Threat: User reads private columns
Page 24

25. © Hortonworks Inc. 2016
Threat: User isn’t Allowed to see Details
Page 25

26. © Hortonworks Inc. 2016
Caution: Shadow Security
Page 26

27. © Hortonworks Inc. 2016
Resources
Page 27

28. © Hortonworks Inc. 2016
Thank You!
Page 28