How to secure your data lake

1 © Hortonworks Inc. 2011–2018. All rights reserved.
How to Secure your Data lake
Omar Ascofare – Hortonworks Solution Engineer
Simon Thibault – Account Executive

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks Company Profile
IPO 4Q14 (NASDAQ: HDP)

Barriers to Real-Time Connected Enterprises
Inability to Leverage “New” and “Traditional” Data Sources
• State of my customers?
• State of my operations?
• State of products in field?
Incomplete View
of Enterprise
Siloed Enterprise
Transaction Data
Consumers
& Customers
Manufacturing
& Supply Chain
Connected Products
& Services
Siloed Real-Time
Data

Traditional Systems Don’t Address Data Diversity Trends
Unstructured Data Machine Data Systems Data

Capture
streaming data
Deliver
perishable insights
Combine
new & old data
Store
data forever
Access
a multi-tenant data lake
Model
with artificial intelligence
DATA AT RESTDATA IN MOTION
ACTIONABLE
INTELLIGENCE
Perishable Insights Historical Insights

6 © Hortonworks Inc. 2011 – 2016. All Rights ReservedPage6 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Hortonworks Influences the
Apache Community
We Employ the Committers
--one third of all committers to the Apache®
Hadoop™ project, and a majority in other
important projects
Our Committers Innovate
and expand both Open Enterprise Hadoop and
Apache NiFi
We Influence the Hadoop Roadmap
by communicating important requirements to the
community through our leaders
A PA C H E H A D O O P C O M M I T T E R S

Hortonworks Strategy - 100% Open Source Data Platform
M A X I M U M C O M M U N I T Y I N N O V AT I O N
T H E
I N N O V AT I O N
A D V A N TA G E
P R O P R I E TA R Y
H A D O O P
T I M E
INNOVATION
O P E N C O M M U N I T Y
Eliminates Risk
of vendor lock-in by delivering
100% Apache open source technology
Maximizes Community Innovation
with hundreds of developers across dozens
of companies
Integrates Seamlessly
through committed co-engineering
partnerships with other leading technologies

Global Data Management With Hortonworks
Globally Manage, Secure, Govern, Consume
DATAPLANE SERVICE (DPS)
MANAGE, GOVERN, SECURE
DATA
LIFECYCLE
MANAGER
DATA
STEWARD
STUDIO*
ISV
SERVICES
*not yet available, coming soon
EXTENSIBLE SERVICES
IBM DSX*CLOUD-
BREAK*
DATA
ANALYTICS
STUDIO*
CONNECTED DATA PLATFORMS
HORTONWORKS
DATA PLATFORM (HDP®) DATA-AT-
REST
HORTONWORKS
DATAFLOW (HDF™)
DATA-IN-MOTION
MODERN DATA USE CASES
EDW
OPTIMIZATION
CYBERSECURITY DATA SCIENCE
ADVANCED
ANALYTICS
HORTONWORKS
CONNECTION
ENTERPRISE SUPPORT
PREMIER SUPPORT
EDUCATIONAL SERVICES
PROFESSIONAL SERVICES
COMMUNITY CONNECTION
HORTONWORKS
PLATFORM SERVICES
OPERATIONAL SERVICES
SMARTSENSE™
DATA
SOURCES
DATA CENTER CLOUD EDGE
Exception
Monitoring
360 View of
Operations
Cyber
Security
Telemetry –
Connected
Devices
Time Series
Sensors,
Control
Systems

What is an Enterprise Data Lake?

P U R P O S E - B U I LT C L U S T E R S
Cluster 1
Application
Security
Governance
Storage
YARN
Operations
Batch
Cluster 2
Application
Security
Governance
Storage
Operations
Interactive
Dedicated
Resource Mgt
…
Application
Security
Governance
Storage
Operations
Real-time
Dedicated
Resource Mgt
Cluster N
An architectural pattern in the data center that uses Hadoop to deliver insight
across a large, broad, diverse set of data at efficient scale. But what is it?
Enterprise Grade
Security, Operations, Governance.
Integration with higher level services
(SAS, SAP, Microsoft, etc..)
Platform for All Data
Store anything, structure/unstructured.
Store everything, retain all attributes.
Land all data in a single place.
Multi-Purpose
Data in a single open platform. Interact in
many ways.
Multi-Tenancy in one shared cluster.
E N T E R P R I S E D ATA L A K E
HDFS Storage Layer
YARN Resource Management
Batch Interactive Real-time In-Memory Search
OPERATIONS
GOVERNANCE
SECURITY
Application Application Application Application Application

Use Case Component Mapping
DATA PLATFORM CORE CAPABILITIES
RESOURCE
MANAGEMENT / SLA
OPERATIONS
AUTHENTICATION / SSO
INFRASTRUCTURE
AUTHORIZATION SCHEDULING
DATA ENCRYPTION /
DATA MASKING
GOVERNANCE / DATA
LINEAGE
BACKUP
BUSINESS TAXONOMY /
META DATA
USE CASES
DATA MANAGEMENT LAYER
RELATIONAL DATA
MODEL
DATA TRANSFORMATION META DATA NoSQL IN MEMORY DATA
PROCESSING
CUSTOM APPLICATION
SINGLE
VIEW OF
ENITY
PREDICTIVE
ANALYTICS
ACTIVE
ARCHIVE
DATA
DISCOVERY
ETL
ONBOARD
END USER TOOLS
BUSINESS INTELLIGENCE DATA SCIENCE
NOTEBOOK
DISCOVERY TOOL MACHINE LEARNING APPLICATION
INTEGRATION
SEARCH
INGESTION
STREAMING
EXTRACT / TRANSFORM /
LOAD
COMMAND LINE
BATCH DATA LOADING
END USER UPLOAD
DATA
ENRICHMENT

Use Case Component Mapping
DATA PLATFORM CORE CAPABILITIES
YARN
AMBARI
KERBEROS
CLOUDBREAK / AMBARI
RANGER OOZIE
RANGER
ATLAS
DLM
ATLAS
USE CASES
DATA MANAGEMENT LAYER
HIVE PIG HCAT HBASE SPARK JAVA
SINGLE
VIEW OF
ENTITY
PREDICTIVE
ANALYTICS
ACTIVE
ARCHIVE
DATA
DISCOVERY
ETL
ONBOARD
END USER LAYER
ODBC / JDBC ZEPPELIN PARTNER OFFERINGS MLLIB WEB SERVICES SOLR
INGESTION
HDF
ETL VENDORS
HDFS
SQOOP
AMBARI
DATA
ENRICHMENT

HDFS Storage Layer
CORE
YARN Resource Management
Storm
INGESTION
Sqoop
HDF
WebHDFS
Kafka
DATAFLOW
NiFi
MR2 Tez Spark HBase Solr
Hive
DATA PROCESSING
Pig Spark Phoenix Storm
Knox Gateway – Perimeter Security
Ranger Authorization / Audit
DATA SECURITY
Kerberos Authentication HDFS Encryption
Cloudbreak
OPS
Ambari
DATA MANAGEMENT & GOVERNANCE
Atlas Metadata FrameworkFalcon Data Lifecycle / Pipeline
SCHEDULING
Oozie Batch Scheduler
HCatalog
Centralized Metadata Management
Click-Stream
SOURCE DATA
Social Data
IoT
Product Data
Sales Data
Enterprise Data
System Administrator
Data Scientist
Data Engineer
OLTP System
EDW
TARGET SYSTEMS
ANALYTIC TOOLS
Tableau /
Excel
SAP / Others
Hive (Tez)
CONSUMPTION
Spark
HBase / Phoenix
Zeppelin
TOOLS
Ambari Views
OPERATIONS
GOVERNANCE
SECURITY
Data Lake Reference Architecture

1402/03/2018
SG DATA OFFER
Big Data As A Service
53
Nodes in Production
6.88T
Storage Capacity
1164
Cores

1502/03/2018
SG DATA
DATALAKE
SOURCE 1 SOURCE 2 SOURCE 3
APPLICATION 1 APPLICATION 2 APPLICATION 3
- Source get the
ownership on the data
- Application requires
access
- Datalake ruled by
conventions
application name, hdfs path, hive
table names,…

What’s the Worst Case Scenario?

Yet we had a solid
roadmap for a data
lake !

No worries boss, I
got this !

How do we Secure the Zoo?

Start with Authentication

Background: Kerberos
⬢ Strongly authenticating and establishing a user’s identity is the basis for secure
access in Hadoop
⬢ Users need to be able to reliably “identify” themselves and have identity
propagated throughout the Hadoop cluster
⬢ Why Kerberos?
⬢ Establishes identity for clients, hosts and services
⬢ Prevents impersonation/passwords are never sent over the wire
⬢ Integrates w/ enterprise identity mgmt tools such as LDAP & Active Directory
⬢ More granular auditing of data access/job execution

22 © Hortonworks Inc. 2011 – 2016. All Rights ReservedPage22 © Hortonworks Inc. 2011 – 2015. All Rights Reserved22

Kerberos Primer
Page 23
Client
KDC
NN
DN
1. kinit - Login and get Ticket Granting Ticket (TGT)
3. Get NameNode Service Ticket (NN-ST)
2. Client Stores TGT in Ticket Cache
4. Client Stores NN-ST in Ticket Cache
5. Read/write file given NN-ST and
file name; returns block locations,
block IDs and Block Access Tokens
if access permitted
6. Read/write block given
Block Access Token and block ID
Client’s
Kerberos Ticket
Cache

Background: HDP + Kerberos
Service
Component
A
Service
Component
B
HDP Cluster
KDC
keytabkeytab
Service
Component
C
keytab
Service
Component
D
keytab
Service
Component
X
Service
Component
X
keytabkeytab
Service
Component
X
keytab
Service
Component
X
keytab
Kerberos is used to
secure the
Components in the
cluster. Kerberos
identities are
managed via
“keytabs” on the
Component hosts.
Principals
for the
cluster are
managed in
the KDC.

Kerberos + Active Directory
Page 25
Cross Realm Trust
Client
Hadoop Cluster
AD /
LDAP KDC
Users: smith@EXAMPLE.COM
Hosts: host1@HADOOP.EXAMPLE.COM
Services: hdfs/host1@HADOOP.EXAMPLE.COM
User Store
Use existing directory
tools to manage users
Use Kerberos tools to
manage host + service
principals
Authentication

Background: Principal and Keytab Generation & Distribution
1. User provides AD Admin Account
credentials to Ambari
2. Ambari connects to AD, creates
principals (Service and Ambari) needed
for cluster
3. Ambari generates keytabs for the
principals
4. Ambari distributes keytabs to Ambari
Server and cluster hosts
5. Ambari discards the AD Admin Account
credentials (optional)
Ambari
Server AD
1 2
4
3
5
HDP
Cluster

Automated Kerberos Setup with Ambari
Ã Wizard driven and automated Kerberos
support
Ã Removes cumbersome, time consuming and
error prone administration of Kerberos
Ã Works with existing Kerberos infrastructure,
including Active Directory to automate
common tasks, removing the burden from
the operator:
• Add/Delete Host
• Add Service
• Add/Delete Component
• Regenerate Keytabs
• Disable Kerberos

Combine Authentication with
Perimeter Security

Authentication—API Security with Knox
• Eliminates SSH “edge node”
• Central API management
• Central audit control
• Service level authorization
• SSO - SAMLv2, Siteminder
and OAM
• LDAP and AD integration
• SSO for Hadoop UIs (Ranger,
Ambari..)
Incubated and led by Hortonworks,
Apache Knox extends the reach of Hadoop REST API without
Kerberos complexities
Integrated with existing IdM
systems
Single, simple point of
access for a cluster
Centralized and consistent
secure API across one or
more clusters
• Kerberos Encapsulation
• Single Hadoop access point
• REST API hierarchy
• Consolidated API calls
• Multi-cluster support

REST API
Hadoop
Services
What does “Perimeter Security” really mean?
Gateway
REST API
Firewall
User
Firewall
required at
perimeter
(today)
Knox Gateway
controls Hadoop
REST API access
through firewall
Hadoop
cluster
mostly
unaffected
Firewall only allows
connections through
specific hosts & ports
forcing requests
through Knox
Knox doesn’t
see or control
what happens
inside the
cluster

3102/03/2018
SG DATA – DATA ACCESS
INSIDE THE CLUSTER
OUTSITE THE CLUSTER
SWEBHDFS
SOLR
KAFKA
OOZIE
APP
User / password
APP
Keytab / Jaas Conf
SSL
SSL

Centralize Administration ,
Authorization & Auditing

Apache Ranger
• Central audit location for all
access requests
• Support multiple destination
sources (HDFS, Solr, etc.)
• Real-time visual query interface
AuditingAuthorization
• Store and manage encryption keys
• Support HDFS Transparent Data
Encryption
• Integration with HSM
• Safenet LUNA
Ranger KMS
• Centralized platform to define, administer
and manage security policies consistently
across Hadoop components
• HDFS, Hive, HBase, YARN, Kafka, Solr,
Storm, Knox, NiFi, Atlas
• Extensible Architecture
• Custom policy conditions, user context
enrichers
• Easy to add new component types for
authorization

Ranger Architecture
HDFS
Ranger Administration Portal
HBase
Hive Server2
Ranger Audit Server
Ranger Plugin
HadoopComponentsEnterprise
Users
Ranger Plugin
Ranger Plugin
Legacy Tools and Data Governance
HDFS
Knox
NifI
Ranger Plugin
Ranger Plugin
SolrRanger Plugin
Ranger Policy Server Integration API
KafkaRanger Plugin
YARNRanger Plugin
Ranger PluginStorm Ranger Plugin Atlas
Solr

Ranger – ABAC Model
⬢ ABAC Model
⬢ Combination of the subject, action,
resource, and environment
⬢ Uses descriptive attributes: AD group,
Apache Atlas-based tags or classifications,
geo-location, etc.
⬢ Ranger approach is consistent with NIST
800-162
⬢ Avoid role proliferation and manageability
issues
Ref : https://nvlpubs.nist.gov/nistpubs/specialpublications/NIST.sp.800-162.pdf

⬢ Comprehensive coverage across Hadoop
ecosystem components
⬢ Plugins for components resident with
component
⬢ Extensible Plugin Model: plugin for
authorizing other sources can be built
Apache Ranger: Comprehensive Extensible Authorization

⬢ Simple Intuitive UI for Policy Editing and
Setup
⬢ Fine-grained specificity by resource type,
user context, tags, and operation
⬢ Supports Access, Tag Based, Dynamic Data
Masking, and Row Filtering Policy Types
Apache Ranger - Intuitive and Granular Policy Management

Apache Ranger Audits - Data Access
⬢ Comprehensive scalable audit logging
⬢ Audits for:
⬢ Resource Access Events with user context
⬢ Policy Edits/Creation/Deletion
⬢ User session information
⬢ Component plugin policy sync operations

Row Level Security in Hive
R A N G E R
Control Access to Rows in Hive Tables based on Context!
Goal: Improve reliability and robustness of HDP by providing Row
Level Security to Hive tables and reducing surface area of security
system
⬢ Capabilities
– Restrict data row access based on
– user characteristics (e.g. group membership) AND
– runtime context
⬢ Use Cases:
v A hospital can create a security policy that allows doctors to view data rows only
for their own patients
v A bank can create a policy to restrict access to rows of financial data based on the
employee's business division, locale or based on the employee's role
v A multi-tenant application can create logical separation of each tenant's each
tenant can see only its data rows.
⬢ Core Technologies: Ranger, Hive
AT L A S
H I V E

Dynamic Data Masking of Hive Columns
R A N G E R
Protect Sensitive Data in real-time with Dynamic Data Masking/Obfuscation!
Goal: Mask or anonymize sensitive columns of data
(e.g. PII, PCI, PHI) from Hive query output
⬢ Benefits
– Sensitive information never leaves database
– No changes are required at the application or Hive layer
– No need to produce additional protected duplicate
versions of datasets
– Simple & easy to setup masking policies
⬢ Core Technologies: Ranger, Hive
AT L A S
H I V E

4102/03/2018
SG DATA – RANGER USAGE
400
Users in Dev Env
10s
Technical Account in
Production
2000+
Ranger Rules

4202/03/2018
SG DATA – RANGER USAGE

Protect your most valuable asset

Data Protection in Hadoop
can be applied at three different layers
in HDP
Storage: encrypt data while it is at rest
Transparent Data Encryption in HDFS, Ranger KMS + HSM, Partner Products
(HPE Voltage, Protegrity, Dataguise)
Transmission: encrypt data over the wire when it
leaves the cluster
SASL (RPC) and TLS (HTTP)
for intracluster communication
Upon Access: apply restrictions when accessed
Ranger (Dynamic Column Masking + Row Filtering), Partner Masking +
Encryption
Data Protection

Data Protection – Layered Approach
ÃEncryption of Data at Rest
– OS Level Encryption (LUKS)
– Certified Partners for volume encryption (e.g: Vormetric (Thales) Protegrity, HPE Voltage Security)
– HDFS TDE file/folder level encryption with keys managed by Ranger KMS, External HSM integration
ÃEncryption of Data on the Wire
– All wire protocols can be encrypted by HDP platform
– Wire-level encryption enhancements (SSL).
ÃGranular Data Protection
– Dynamic Masking + Row Filtering for Hive with Ranger
– Classification Based Security with Ranger + Atlas
– Element level encryption/masking from certified partners (HPE Voltage, Protegrity)

Ranger KMS
Transparent Data Encryption in HDFS
NN
A B
C D
HDFS Client
A B
C D
A B
C D
DN DN DN
Benefits
v Selective encryption of relevant files/folders
v Prevent rogue admin access to sensitive data
v Fine grained access controls
v Transparent to end application w/o changes
v Ranger KMS integrated to external HSM
(Safenet Luna) adding to reliability/security of
KMS
SafeNet-
Luna HSM

Integrate Governance & Security

STRUCTURED
Atlas: Open Metadata & Governance Services
TRADITIONAL
RDBMS
METADATA
MPP
APPLIANCES
Kafka Storm
Sqoop
Hive
ATLAS
METADATA
HBase
RANGER
HDFS
Partners
Provides metadata-driven core foundational governance
services for Hadoop and enterprise data ecosystem
Data Lineage/Provenance
• Captures data lineage across components
Data Classification
• Supports classification of data assets using tags –
PII, PHI, PCI, EXPIRES_ON, CLAIMS,
LIFE_INSURANCE
Metadata Catalog Search
• Free text search on metadata
• Advanced search using DSL
Integrations
• OOtB real-time metadata and lineage ingestion
with Hive, Sqoop, Storm/Kafka
• APIs for custom metadata ingestion
• Apache Ranger integration for classification based
security
Metadata Repository
• Flexible metamodel to capture technical, business,
operational metadata
• Out-of-box models for Hive, Storm, Sqoop, HDFS,
Kafka, HBase
• APIs to register custom models

Background: DGI Community becomes Apache Atlas
May
2015
Apache
Atlas
Incubation
DGI group
Kickoff
Dec
2014
Apr
2017
HDP 2.6/
Apache 0.8
Release
Global Financial
Company
* DGI: Data Governance Initiative
Aug
2016
HDP 2.5/
Apache 0.7
Foundation
Release
Apache 0.8/HDP 2.6
• Simplified Search UI
• Simplified APIs
• Classification-based security for HDFS,
Kafka, HBase
• Knox SSO
• Performance/scalability improvements
Apache 0.7.1/HDP 2.5.3
• High availability support
• LDAP Authentication/Authorization
• Classification based security for Hive
• UI Redesign
• #Committers – 35
• Code contributors from
- Hortonworks, IBM, Aetna,
Merck, Target

High Level Architecture: 4 Key points
Type System
Repository
Search DSL
Bridge
Hive Storm
Solr Custom
REST API
Graph DB
Search
Kafka
Sqoop
Connectors
Messaging Framework
3 REST API
Modern, flexible access
to Atlas services, HDP
components, UI &
external tools
1 Data Lineage
Only product that
captures lineage
across Hadoop
components at
platform level.
4 Exchange
Leverage existing
metadata / models by
importing it from
current tools. Export
metadata to
downstream systems
2 Agile Data Modeling:
Type system allows
custom metadata
structures in a
hierarchy taxonomy

Scalable: Dynamic Tag-based Access Policy

Scalable: Dynamic Tag-based Access Policy
Key Benefits:
• New scalable metadata based security
paradigm
• Dynamic, real-time policy
• Active protection – fast updates to changes
• Centralized and simple to manage policy

Tag based Policies in Ranger
R A N G E R
Control Access to Resources based on Classification (Tagging)
Goal: Separation of resource-classification from access
authorization
⬢ Capabilities
– Authorize data access based on resource classification (e.g.
sensitive data such as PII, PCI) rather than resource type itself
⬢ Benefits:
v After tagging, authorization for tag is automatically enforced
=> no need to create or update policies for the resource
v Single authorization policy for a tag across various Hadoop
components
⬢ Core Technologies: Ranger + Atlas (for tagging)
⬢ Available for: Hive, HDFS, Yarn, HBase, Kafka, Storm,
Solr, Knox
AT L A S H I V EH B A S EH D F S
S TO R M
AT L A SK A F K AYA R N
AT L A S
K N OX
S O L R

Apache Atlas: Connectors and Ecosystem
Custom
Integration
PartnerPartner
Apache Atlas
RDBMS
Apache
Kafka
Pending:

Lineage
• Where does this data originate from (source/provenance)?
• Upstream path: Path through all data assets and processes leading up to current data asset
Impact
• How is this data being used ?
• What other data assets (derivative/dependent) does this impact?
• Downstream path: Path through all data assets and processes leading out of current data asset
Used for forensics
• Impact analysis
• Auditing and Compliance
Apache Atlas : Lineage

Apache Atlas: Lineage and Impact

Apache Atlas Classification : use case – access expiry
Data expiration
• EXPIRES_ON classification with attribute expiry_date
• tax_2009 table tagged with EXPIRES_ON(expiry_date=2016/12/31)
• tax_2010 table tagged with EXPIRES_ON(expiry_date=2017/12/31)
• Apache Ranger policies use the attribute to block access after expiry date

Apache Atlas Classification : use case – REFERENCE_DATA
Security policy enforcement for denying updates on immutable data
assets
• REFERENCE_DATA classification associated with immutable hive_table eu_countries
• Apache Ranger policies block updates on the table for all users except admins

Apache Atlas Classification: use case – attribute based authorization
Data quality
• Deny access to analysts group based on data quality threshold

Apache Atlas Classification: use case - hierarchy
Security policy enforcement based on classification hierarchy
• Data assets classified as PII will be denied for all contractors
• Data assets classified as FinancePII will be denied for anyone not in
Finance group
• Data assets classified as VendorPII will be denied for anyone not in
Partner Manager group
• ..hence contractors will be denied access to data assets classified as PII,
FinancePII, and VendorPII
PII
Finance
PII
Vendor
PII
Deny: contractors
Deny: public
Except: Finance
Deny: public
Except: PartnerManager

Metadata Catalog Search : Free Text
Search for a hive_table classified as ‘PII’ and name starting with ‘emp’
Filter by
Data Asset type
Filter by
Classification
Search text
Wildcards: emp*, *dept*
Logical expressions: emp* AND *dept*

Metadata Catalog Search : Advanced
Filter by
Data asset type
Search for a hive_table named ‘employees’ and owner ‘hive’
DSL search with SQL like syntax
Select columns from impressions table in raw database
hive_column where table.name=‘impressions’ and table.db.name = ‘raw’
DSL query string

6302/03/2018
SG DATA – JOURNEY TO DATA GOVERNANCE TOOLING
SG | Data
Catalog
2015 2017
Vendors POC
2018
- Data Quality
- Data Lineage
- Fetch Metastore
metadata
- Expose datasets in
single dashboard
SG | Data
CatalogAS A CENTRAL
PIECE CONNECTED
WITH EVERY TOOLS

Wrap-up

Protecting the Elephant in the Castle…..
Kerberos,
Wire Encryption
HDFS Encryption
Apache Ranger
Network Segmentation,
Firewalls
LDAP/AD
Apache Knox

HDFS
Typical Flow – SQL Access through Beeline client
HiveServer 2
A B C
Beeline
Client

HDFS
Typical Flow – Authenticate through Kerberos
HiveServer 2
A B C
KDC
Login into Hive using
AD password
Hive gets
Namenode
(NN) service
ticket
Hive creates
map reduce
using NN ST
Client gets
service ticket for
Hive
Beeline
Client
Active
Directory

HDFS
Typical Flow – Add Authorization through Ranger
HiveServer 2
A B C
KDC
Hive gets
Namenode
(NN) service
ticket
Column level
access control,
auditing
Ranger
Beeline
Client
File level
access control
Active
Directory
Import
users/groups
from LDAP
Login into Hive using
AD password

HDFS
Typical Flow – Firewall, Route through Knox Gateway
HiveServer 2
A B C
KDC
Use Hive ST,
submit query
Hive gets
Namenode
(NN) service
ticket
Hive creates
map reduce
using NN ST
Ranger
Knox gets
service ticket for
Hive
Knox runs as proxy
user using Hive ST
Original
request w/user
id/password
Client gets
query result
Beeline
Client
Apache
Knox
Active
Directory

HDFS
Typical Flow – Add Wire and File Encryption
HiveServer 2
A B C
KDC
Use Hive ST,
submit query
Hive gets
Namenode
(NN) service
ticket
Hive creates
map reduce
using NN ST
Ranger
Knox gets
service ticket for
Hive
Knox runs as proxy
user using Hive ST
Original
request w/user
id/password
Client gets
query result
SSL
Beeline
Client
SSL SASL
SSL SSL
Apache
Knox
Active
Directory

HDP Security: Comprehensive, Complete, Extensible
Data Protection
Protect data at rest and in motion
Audit
Maintain a record of data access
Authorization
Provision access to data
Authentication
Authenticate users and systems
Administration
Central management and consistent security
Single administrative console to set policy across the
entire cluster: Apache Ranger
Authentication for perimeter and cluster; integrates
with existing Active Directory and LDAP solutions:
Kerberos | Apache Knox
Consistent authorization controls across all Apache
components within HDP: Apache Ranger
Record of data access events across all components
that is consistent and accessible: Apache Ranger
Secure data in motion and data at rest: HDFS TDE w/
Ranger KMS + HSM, Ranger Data Masking + Row
Filtering, Wire encryption + Partner Solutions

What’s next !?

Hortonworks Platform Security – What Do We Do Today?
• We closely follow and track CVE’s via Apache CVE process e.g.
• https://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-33597/Apache-Ranger.html
• https://cwiki.apache.org/confluence/display/RANGER/Vulnerabilities+found+in+Ranger
• Our security SME team monitors CVEs and files appropriate JIRAs against specific components for
remediation via internal HDPSEC project (not publicly exposed due to sensitive nature of info)
• We run Coverity code scans on Apache projects frequently and remediate any critical vulnerabilities found
• We have a process for customers to report critical security vulnerabilities, we create security patches as
needed
• Release notes in our public documentation contain a section on CVE or vulnerabilities fixed in a release
• Tech Alerts are sent to customers upon CVE fixes with patch instructions, upgrade vehicles and/or
maintenance releases
• HP Fortify scans are built into the release engineering process for each release
• Dynamic application testing such as pen testing is done via various tools for each release (BurpSuite, Nikto,
nmap, ZAP Scanning)

7402/03/2018
SG DATA – CUSTOM MIDDLEWARE
SG DATA CONSOLE

Questions?

Thank you

How to secure your data lake

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to secure your data lake

Similar to How to secure your data lake (20)

Recently uploaded

Recently uploaded (20)

How to secure your data lake