Biren Saini talked to us about Apache Ranger on 10/27/15

1. Hadoop Data Security
with
Apache Ranger
Biren Saini
© Hortonworks Inc. 2011 – 2015. All Rights Reserved

2. Page 2 © Hortonworks Inc. 2015
About me
• Biren Saini
• Senior Solutions Engineer
• Governance SME Lead
• Overall 15 years of technology experience
@ Hortonworks}

3. Page 3 © Hortonworks Inc. 2015
Agenda
• Hadoop Security Overview
• Apache Ranger
– Introduction
– Architecture
– Sample Flow
– Best Practices
– Ranger Stacks
– Demo

4. Page 4 © Hortonworks Inc. 2015
Overview of Security in Hadoop

5. Page 5 © Hortonworks Inc. 2015
5 Pillars of Security
• Authentication
• Authorization
• Audit
• Encryption
• Centralized Administration

6. Page 6 © Hortonworks Inc. 2015
Security Tools in Hadoop world
• Kerberos (authentication)
• Apache Knox (authentication)
• AD/LDAP (authentication)
• Apache Ranger (authorization, audit, kms)
• HDFS TDE (data encryption)
• Wire Encryption (data protection)

7. Page 7 © Hortonworks Inc. 2015
HDFS
Typical Flow – SQL Access through Beeline client
HiveServer 2
A B C
Beeline
Client

8. Page 8 © Hortonworks Inc. 2015
HDFS
Typical Flow – Authenticate through Kerberos
HiveServer 2
A B C
KDC
Login into Hive using
AD password
Hive gets
Namenode (NN)
service ticket
Hive creates
map reduce
using NN ST
Client gets
service ticket for
Hive
Beeline
Client
Active
Directory

9. Page 9 © Hortonworks Inc. 2015
HDFS
Typical Flow – Add Authorization through Apache Ranger
HiveServer 2
A B C
KDC
Hive gets
Namenode (NN)
service ticket
Column level
access control,
auditing
Ranger
Beeline
Client
File level
access control
Active
Directory
Import users/groups from
LDAP
Login into Hive using
AD password

10. Page 10 © Hortonworks Inc. 2015
HDFS
Typical Flow – Firewall, Route through Knox Gateway
HiveServer 2
A B C
KDC
Use Hive ST,
submit query
Hive gets
Namenode (NN)
service ticket
Hive creates
map reduce
using NN ST
Ranger
Knox gets
service ticket for
Hive
Knox runs as proxy
user using Hive ST
Original
request w/user
id/password
Client gets
query result
Beeline
Client
Apache
Knox
Active
Directory

11. Page 11 © Hortonworks Inc. 2015
HDFS
Typical Flow – Add Wire and File Encryption
HiveServer 2
A B C
KDC
Use Hive ST,
submit query
Hive gets
Namenode (NN)
service ticket
Hive creates
map reduce
using NN ST
Ranger
Knox gets
service ticket for
Hive
Knox runs as proxy
user using Hive ST
Original
request w/user
id/password
Client gets
query result
SSL
Beeline
Client
SSL SASL
SSL SSL
Apache
Knox
Active
Directory

12. Page 12 © Hortonworks Inc. 2015
Apache Ranger

13. Page 13 © Hortonworks Inc. 2015
Apache Ranger
• Provides centralized policy definition for authorizing & auditing access
to resources in a consistent manner.
• Supported components as of v0.5
• HDFS
• HBase
• Hive
• YARN
• Knox
• Storm
• Solr
• Kafka

14. Page 14 © Hortonworks Inc. 2015
Setup Authorization Policies
14
file level
access
control,
flexible
definition
Control
permissions

15. Page 15 © Hortonworks Inc. 2015
Monitor through Auditing

16. Page 16 © Hortonworks Inc. 2015
Agent AgentAgent AgentAgent Agent
Apache Ranger authZ Architecture
HBase Hive YARN Knox Storm Solr Kafka
Agent
HDFS
Agent
Administration Portal
(Ranger UI)
REST APIs
DB
SOLR
HDFS
Policy
Server
LDAP/AD
user/group
sync
Log4j
KMS
Audit
Server
User Sync
Server

17. Page 17 © Hortonworks Inc. 2015
Hadoop Cluster
Sample Simplified Workflow - HDFS
Policy
Server
Audit
Server
Administration Portal
Agent
Namenode
Audit
Store
Ranger
Policy
Store
Unauthorized user attempts
to access the data
User access is denied
No Policy defined.

18. Page 18 © Hortonworks Inc. 2015
Hadoop Cluster
Sample Simplified Workflow - HDFS
Policy
Server
Audit
Server
Administration Portal
Admin sets policies for HDFS files/
folder1a
1b
1d
Agent
Namenode
Audit
Store
Ranger
Policy
Store
1c

19. Page 19 © Hortonworks Inc. 2015
Hadoop Cluster
Sample Simplified Workflow - HDFS
Policy
Server
Audit
Server
Administration Portal
Admin sets policies for HDFS files/
folder1a
Data scientist runs a map
reduce job
User
Application
Analysts access HDFS
data through application
IT users access
HDFS through CLI
1b
2a
2a
2a
Agent
Namenode
Namenode provides
resource access to
user/client
Namenode uses
Agent for Authorization2b
Audit
Store
2d
2c
Ranger
Policy
Store
1d
1c

20. Page 20 © Hortonworks Inc. 2015
Hadoop Cluster
Sample Simplified Workflow - HDFS
Policy
Server
Audit
Server
Administration Portal
Admin sets policies for HDFS files/
folder1a
Data scientist runs a map
reduce job
User
Application
Analysts access HDFS
data through application
IT users access
HDFS through CLI
1b
2a
2a
2a
Agent
Namenode
Namenode provides
resource access to
user/client
Namenode uses
Agent for Authorization2b
Audit
Store
Admin requests the Audit report3a
3b
3c
2d
2c
Ranger
Policy
Store
1d
1c

21. Page 21 © Hortonworks Inc. 2015
Ranger UserSync Best Practice
21
• Ensure LDAPS is used to integrate with Ranger
• Create OU ONLY for Hadoop users for performance
• Only run usersync when necessary
– How much users are being added and how often
– How much users are changing roles
– Too much syncing can degrade LDAP performance
• Do not sync anonymously

22. Page 22 © Hortonworks Inc. 2015
Ranger Audit Best Practices
22
• HDFS
– Long term storage that can be used to understand user event
trends and predict anomaly
• RDBMS
– When SQL is preferred by auditors
– MySQL, Oracle, Postgres, SQL Server
• Solr
– Nice quick reporting metrics to understand user event trends
• Log4j Appenders

23. Page 23 © Hortonworks Inc. 2015
Ranger Stacks
• Apache Ranger v0.5 supports stack-model to enable easier onboarding
of new components, without requiring code changes in Apache Ranger.
Ranger Side Changes
Define Service-type
Secured Components Side Changes
Develop Ranger Authorization Plugin
• Create a JSON file with
following details :
- Resources
- Access types
- Config to connect
• Load the JSON into Ranger.
• Include plugin library in the secure component.
• During initialization of the service: Init
RangerBasePlugIn & RangerDefaultAuditHandler class.
• To authorize access to a resource: Use
RangerAccessRequest.isAccessAllowed()
• To support resource lookup: Implement
RangerBaseService.lookupResource() &
RangerBaseService.validateConfig()
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741207

24. Page 24 © Hortonworks Inc. 2015
Summary & Misc. points
24
• All functions are available as Rest API
• Ranger integrates with AD/LDAP for ranger login as well as user sync.
• Support for High Availability (HA)
• Support for Transparent Data Encryption with KMS implementation
• Tighter integration with Apache Ambari
• Stack based implementation of Plugins
• Ranger also has the KMS for HDFS TDE.
• Some features in development are
– Spark support
– Time based authorization
– Geo Location based authorization

25. Page 25 © Hortonworks Inc. 2015
Demo - HDFS
Admin
SamTom
/demo/data/trihug
/demo/data/trihugRanger UI
WRITE Access denied READ Access denied
1
2
SamTom
/demo/data/trihug
WRITE Access allowed READ Access allowed
3
Grants access
READ for Sam
WRITE for Tom
Ranger Plugin
gets the update
WRITE Access denied
hdfs:hdfs rwx --- ---
Elevated Privileges Restricted Privileges
Directory already exists

26. Page 26 © Hortonworks Inc. 2015
Demo - Hive
Admin
SamTom
tickers
eod
Ranger UI
WRITE Access denied READ Access denied
1
2
SamTom
WRITE Access allowed READ Access to SOME
COLUMNS allowed
3
Grants access
READ for Sam
ALL for Tom
Ranger Plugin
gets the update
WRITE Access denied
hive tables
tickers
eod hive tables
tickers
eod hive tables
SOME COLUMNS
READ Access to ALL
COLUMNS denied
Created by “hive” user in
trihug schema
Elevated Privileges Restricted Privileges
GRANT Access allowed
DB already exists

27. Page 27 © Hortonworks Inc. 2015
Demo time..

28. Page 28 © Hortonworks Inc. 2015
Thank you.
Questions?