Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Page1 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Securing Hadoop With Apache Ranger
Strategies & Best Practices
1...
Page2 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Session overview
Secured by:
Page3 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Speakers Introduction
Selvamohan Neethiraj
Senior Director, Ente...
Page4 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Agenda
Current
Hadoop
Security
What
Ranger
Brings
Ranger
Demo
Be...
Page5 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Universal Security Principles
• Authentication
– Who is the user...
Page6 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
How Hadoop handles security?
• Authentication [Simple, Kerberos]...
Page7 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Security on Hadoop tool sets
• Hdfs
• Hive
• Hbase
• Knox
• Stor...
Page8 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Agenda
Current
Hadoop
Security
What
Ranger
Brings
Ranger
Demo
Be...
Page9 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Example Access Scenario
HDFS
HDFS
ACL
HDFS
audit log
Users
Acces...
Page10 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Example Access Scenario with Ranger
HDFS
HDFS
ACL
Users
Access ...
Page11 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Core Ranger Features:
• Ranger 0.4.0
• Authorization Support fo...
Page12 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger: Centralized Policy Admin
Page13 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Architecture
Ranger
Policy Admin Server
Ranger
User Sync...
Page14 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Apache Ranger Summary
• Comprehensive security for Hadoop ecosy...
Page15 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Implementation - components
 Central interface for secu...
Page16 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Policy Admin Server
• Provides web interface to support ...
Page17 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger User/Group Sync Server
• Stand-alone Java server
• Retri...
Page18 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – HDFS Integration
• HDFS Permissions
• POSIX lik...
Page19 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – HDFS Integration
• Ranger Plugin acts as an aut...
Page20 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – Hive Integration
• Hive facilitates querying an...
Page21 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – Hive Integration
• Ranger Plugin acts as author...
Page22 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – Hive Integration
• When the user executes GRANT...
Page23 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – HBase Integration
• Hbase is a non-relational D...
Page24 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – HBase Integration
• Ranger Plugin is implemente...
Page25 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – HBase Integration
• Similar to Hive, Hbase also...
Page26 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – Knox Integration
• Knox provides perimeter secu...
Page27 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – Knox Integration
• Knox provides service level ...
Page28 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – Storm Integration
• Storm is a distributed real...
Page29 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Plugin – Storm Integration
• Ranger Plugin acts as an au...
Page30 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Auditing
• Auditing is configured through policies (like...
Page31 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Agenda
Current
Hadoop
Security
What
Ranger
Brings
Ranger
Demo
B...
Page32 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Demo
Page33 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Agenda
Current
Hadoop
Security
What
Ranger
Brings
Ranger
Demo
B...
Page34 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Strategies & Best practices
• Authentication Process
• Kerberos...
Page35 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ranger Resources
• Do NOT Miss BoF on Apache Ranger/Knox
• Inte...
Page36 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
References
• http://hadoop.apache.org/docs/current/hadoop-proje...
Page37 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Q & A
Page 37
Upcoming SlideShare
Loading in …5
×

Securing Hadoop with Apache Ranger

18,830 views

Published on

Hadoop Summit 2015

Published in: Technology
  • Have u ever tried external professional writing services like ⇒ www.HelpWriting.net ⇐ ? I did and I am more than satisfied.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • ⇒ www.HelpWriting.net ⇐ is a good website if you’re looking to get your essay written for you. You can also request things like research papers or dissertations. It’s really convenient and helpful. If you’ve got something very last-minute then it can be a little risky but either way it’s probably better than anything you can throw together :).
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Read this heartfelt letter below from Sonasi Samita, a disease-ridden man stricken with kidney failure, diabetes, gout, heart problems, and blindness. He tells his tear-jerking story on how the Demolisher system has totally changed his life! Sonasi says he's convinced that the Demolisher system is God's answer to his prayers! ♥♥♥ http://t.cn/A6zP2GDT
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Follow the link, new dating source: ♥♥♥ http://bit.ly/2ZDZFYj ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area is here: ❶❶❶ http://bit.ly/2ZDZFYj ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Securing Hadoop with Apache Ranger

  1. 1. Page1 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Securing Hadoop With Apache Ranger Strategies & Best Practices 11 June 2015 – Hadoop Summit, San Jose Presented by: Selvamohan Neethiraj, Hortonworks Velmurugan Periasamy, Hortonworks
  2. 2. Page2 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Session overview Secured by:
  3. 3. Page3 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Speakers Introduction Selvamohan Neethiraj Senior Director, Enterprise Security Development, HortonWorks Velmurugan Periasamy Senior Technical Manager, HortonWorks
  4. 4. Page4 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Agenda Current Hadoop Security What Ranger Brings Ranger Demo Best Practices Q & A
  5. 5. Page5 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Universal Security Principles • Authentication – Who is the user? • Authorization – What can the user do? • Auditing – Record user’s activities • Data Integrity – It is what it is • Confidentiality – Restricted
  6. 6. Page6 © Hortonworks Inc. 2011 – 2015. All Rights Reserved How Hadoop handles security? • Authentication [Simple, Kerberos] • Authorization [Acl on files/folders] • Auditing [audit log on hosts] • Data Integrity [SSL communication] • Confidentiality [TDE]
  7. 7. Page7 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Security on Hadoop tool sets • Hdfs • Hive • Hbase • Knox • Storm • Oozie • Spark • Kafka • Solr • KMS
  8. 8. Page8 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Agenda Current Hadoop Security What Ranger Brings Ranger Demo Best Practices Q & A
  9. 9. Page9 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Example Access Scenario HDFS HDFS ACL HDFS audit log Users Access HDFS file/folder HDFS checks its ACL HDFS Writes audit Hive Hive Authoriza tion Hive audit log Hive authz check Hive Writes audit Access Hive table HBase HBase ACL Hbase audit log Hbase authz check HBase Writes audit Access Hbase table
  10. 10. Page10 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Example Access Scenario with Ranger HDFS HDFS ACL Users Access HDFS file/folder Plugin Enforces policies And writes audit Hive Hive Authoriza tion Plugin Enforces policies And writes audit Plugins Write to Centralized Audit store Access Hive table HBase HBase ACL Plugin Enforces policies And writes audit Access Hbase table Ranger Policy store Ranger Audit store Ranger Plugin Ranger Plugin Ranger Plugin Plugins Download Defined Policies from Centralized Policy store Centralized Security Policy Management and Auditing provided by Ranger
  11. 11. Page11 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Core Ranger Features: • Ranger 0.4.0 • Authorization Support for HDFS, Hive, HBase, Knox, Storm • Audit destination support for HDFS, DB • Ranger 0.5.0 (Released yesterday!) • Support for High Availability (HA) • Support for Transparent Data Encryption with KMS implementation • Support for more plugins (Yarn, Kafka, Solr) • Solr as audit destination • Stack based implementation of Plugins • Dynamic policies with many other attributes • Internal permission model within Ranger Admin UI • Tighter integration with Apache Ambari
  12. 12. Page12 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger: Centralized Policy Admin
  13. 13. Page13 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Architecture Ranger Policy Admin Server Ranger User Sync Server HDFS Ranger Plugin Enterprise Directory Services Sync Users Security Administrators Enterprise Users Define Policies Audit user activity Sync Users Enterprise Legacy Systems Policy Integration Access Data in Hadoop Systems Authenticate Ranger Plugins Authorize the Access & Audit the Activity Hive Ranger Plugin HBase Ranger Plugin Knox Ranger Plugin Storm Ranger Plugin Ranger Audit Store Sync Policies Store Audit Activity Ranger Policy DB
  14. 14. Page14 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Apache Ranger Summary • Comprehensive security for Hadoop ecosystem • Apache Incubator Project – 100% open source • Centralized Administration of Security Policies • Integration with HDFS, Hive, Hbase, Knox, Storm, … • More components integration coming • Ensures consistent coverage across entire Hadoop stack
  15. 15. Page15 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Implementation - components  Central interface for security administration.  Users can create and update policies, view audit activities, manage users Ranger policy admin server  Synchronization utility to pull users and groups from Unix or LDAP or Active Directory.  User/group information is stored within Ranger admin policy DB and used for policy definition. Ranger user sync server  Lightweight Java programs within Hadoop component.  Pull in policies from policy admin server and store them locally in cache.  Act as Authorization module and evaluate user requests against security policies before granting access.  Collect data from user request and store this data into the audit store. Ranger plugins
  16. 16. Page16 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Policy Admin Server • Provides web interface to support ranger activities • Define repositories (resources to be authorized) • Define access policies (specify which users/groups can access to which resources) • Manage users/groups • Define auditing policies • View/Analyze audit data • Runs embedded tomcat server • Supports LDAP/AD and Unix authentication
  17. 17. Page17 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger User/Group Sync Server • Stand-alone Java server • Retrieves users/groups from enterprise directories • Creates these users/groups in Ranger DB for.. • supporting policy definition • allowing access to Ranger policy admin server • Supports Synchronization of users/groups: • LDAP • Active Directory • Unix
  18. 18. Page18 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – HDFS Integration • HDFS Permissions • POSIX like permission model (owner/group for files and folders) • ACL’s for fine-grained permissions (for specific set of users/groups) • hdfs dfs -getfacl [-R] <path> • hdfs dfs -setfacl [-R] [options] <path> • dfs.permissions.enabled must be set to true
  19. 19. Page19 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – HDFS Integration • Ranger Plugin acts as an authorizer within Namenode. • Need to install in all name nodes (in HA environment) • User can define policies on files and folders • Use of wildcard to define policies (/finance/audit_*) • Read, Write, Execute permissions are allowed • Plugin evaluates HDFS requests and provide access • If no specific ranger policy exists, HDFS ACLs are used as fallback
  20. 20. Page20 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – Hive Integration • Hive facilitates querying and managing large datasets in distributed storage (on top of Hadoop) • HiveServer2 (HS2) is a server interface to Hive • HiveServer2 supports Access Control similar to relational database model • SELECT/UPDATE/DELETE permissions on tables/columns • Permission defined for USERS/ROLES • Also provides a pluggable authorizer model
  21. 21. Page21 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – Hive Integration • Ranger Plugin acts as authorization provider for Hive Server2 • User can define policies on databases, tables/view, columns and UDFs • Select, Update, Create, Drop, Alter, Index, Lock Permissions allowed • Plugin evaluates Hive requests and grants/denies access based on the policies and creates necessary audit logs based on audit • Specific Ranger policy must exist for gaining access
  22. 22. Page22 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – Hive Integration • When the user executes GRANT/REVOKE statements, the hive plugin will creates/deletes necessary Ranger Policies (provided the user has permission to create/delete policies) • GRANT SELECT, UPDATE, ALTER ON TABLE test_data TO USER guest; ← will create a new Ranger policy • REVOKE UPDATE, ALTER ON TABLE test_data FROM USER guest; ← will update/delete existing Ranger policy • Also, ROLE specified in the GRANT/REVOKE statements will be mapped to corresponding GROUPS from your corporate directories • Optionally, you can disable GRANT/REVOKE commands i.e. to force authorization policies management solely via Ranger Policy Admin
  23. 23. Page23 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – HBase Integration • Hbase is a non-relational DB on top of Hadoop/HDFS • HBase Provides Role Based Access Control/ACLs • ACLs are implemented as a coprocessor called AccessController • Ranger implements a similar coprocessor for enforcing access control based on Ranger Policies
  24. 24. Page24 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – HBase Integration • Ranger Plugin is implemented as a coprocessor of HBase Master/Region Servers to enforce Ranger Policies • User can define policies on tables, column families and qualifiers • Supports wildcard in defining policies (table = fin_*, col_fam = audit*) • Read, Write, Create, Admin permissions allowed • Plugin evaluates Hbase requests and grants/denies access based on the policies and creates necessary audit logs based on audit • Specific ranger policy must exist for gaining access
  25. 25. Page25 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – HBase Integration • Similar to Hive, Hbase also supports Grant/Revoke commands to manage access control from within Hbase shell. • Ranger HBase plug-in creates/updates Ranger policies to reflect permissions set via Grant/Revoke (from within Hbase) • grant ‘bob’, ‘RWCA’, ‘test_data’ ← will create a new Ranger policy • grant ‘bob’, ‘R’, ‘test_data’ ← will update the Ranger policy created by earlier grant; • Option to disable Grant/revoke commands i.e. to force authorization policies management solely via Ranger Policy Admin tool
  26. 26. Page26 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – Knox Integration • Knox provides perimeter security for Hadoop REST API • Authentication and token verification at the perimeter • Authentication integration with enterprise and cloud identity management systems • Service level authorization at the perimeter • Single URL hierarchy that aggregates REST APIs of a Hadoop cluster • Hadoop services with built-in support currently! • WebHDFS, WebHCat, Oozie • Hbase, Hive, Yarn
  27. 27. Page27 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – Knox Integration • Knox provides service level authorization based on xml configuration. • Ranger Knox Plugin will allow service level authorization enforcement via Ranger Policies by acting as Authz Provider within Knox Gateway • User can define policies on topologies and services • Provide access to services based on user/group/ip-address • E.g. Finance group will have access to WebHDFS from 10.1.1.* • Plugin evaluates Knox requests and grants/denies access based on the policies and creates necessary audit logs based on audit policies
  28. 28. Page28 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – Storm Integration • Storm is a distributed real-time computation system. • Storm provides general primitives for real-time computation similar to how Hadoop provides general primitives for batch processing. • Storm Topologies :: Hadoop MR Jobs • Nimbus server (runs on master node) and Supervisor (runs on each worker node). Communication via ZooKeeper • Storm security is implemented based on Kerberos authentication
  29. 29. Page29 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Plugin – Storm Integration • Ranger Plugin acts as an authorizer within Nimbus server • Ranger plugin then can authorize all incoming requests based on the Ranger Policies. • User can define policies on topologies • Permissions that can be set • getClusterInfo, Submit/Get/Kill/Activate/Deactivate Topology • Plugin evaluates Storm requests and grants/denies access based on the policies and creates necessary audit logs based on audit policies
  30. 30. Page30 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Auditing • Auditing is configured through policies (like authorization) • Resource access audit supported with following attributes • User id, request type, repository, access resource, IP address, timestamp, access granted/denied • Admin audit • Changes to policies, login sessions and plugin-monitoring • Audit Destination - HDFS, SOLR are supported, along with custom LOG4J based logging …
  31. 31. Page31 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Agenda Current Hadoop Security What Ranger Brings Ranger Demo Best Practices Q & A
  32. 32. Page32 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Demo
  33. 33. Page33 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Agenda Current Hadoop Security What Ranger Brings Ranger Demo Best Practices Q & A
  34. 34. Page34 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Strategies & Best practices • Authentication Process • Kerberos is MUST • Centralized Access Control & Auditing • Use Ranger to define & enforce the security policies • Enable Wire encryption • For confidential data, use Hadoop TDE for data encryption (at rest) • Aware of file permission for keystore & kerberos keytab • Protect access to log files
  35. 35. Page35 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Ranger Resources • Do NOT Miss BoF on Apache Ranger/Knox • Interested in Security ? -- Join Ranger Community to strengthen Hadoop Security!! • Join and Contribute! -- Apache Ranger website - http://ranger.incubator.apache.org/ -- Ranger Wiki - https://cwiki.apache.org/confluence/display/RANGER/Index
  36. 36. Page36 © Hortonworks Inc. 2011 – 2015. All Rights Reserved References • http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop- hdfs/HdfsPermissionsGuide.html • https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Authorization • http://hbase.apache.org/book/hbase.accesscontrol.configuration.html • https://knox.apache.org/books/knox-0-5-0/knox-0-5-0.html • https://github.com/apache/storm/blob/master/SECURITY.md
  37. 37. Page37 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Q & A Page 37

×