Hadoop Security
Architecture
Owen O’Malley
oom@yahoo-inc.com

Hadoop
Outline
•
•
•
•
•
•

Problem Statement
Security Threats
Solutions to Threats
HDFS
MapReduce
Oozie

• Interfaces
• Performa...
Problem Statement
• The fundamental goal of adding Hadoop security is that Yahoo's data
stored in HDFS must be secure from...
Communication and Threats

User
Process

Job
Tracker

Name
Node

Task
Tracker

Oozie

Data
Node

Task

NFS

MObStor

ZooKe...
Security Threats in Hadoop
•

User to Service Authentication
– No User Authentication on NameNode or JobTracker
•

Client ...
Solutions to Threats
• Add Kerberos-based authentication to NameNode
and JobTracker.
• Add delegation tokens to HDFS and s...
Out of Scope for 0.20.100
• Protecting against root on slave nodes:
– Encryption of RPC messages
– Encryption of block tra...
HDFS Security
• Users authenticate via Kerberos
• MapReduce jobs can obtain delegation
tokens for later use.
• When client...
HDFS Authentication
ke r

)
b(joe

Application

Name
Node

delg(jo
e)

MapReduce
Task

kerb(hdfs)
bloc

k to

ken

Data
No...
What does this *really* look like?
• Need a Kerberos ticket to work
– kinit –l 7d oom@DS.CORP.YAHOO.COM
– hadoop fs –ls
– ...
Kerberos Dataflows
Key
Distribution
Center

Get TGT

Request Service Ticket

Name Node
hdfs/host@YGRID

User
joe@DS.CORP

...
Delegation Token
• Advantages over using Kerberos directly:
– Don’t trust JobTracker with credentials
– Avoid MapReduce ta...
Block Access Token
• Only NameNode knows the set of users allowed to
access a specific block, so the NameNodes gives
an au...
MapReduce Security
• Require Kerberos authentication from client.
• Secure the information about pending and running jobs
...
MapReduce Authentication
Application

kerb(joe)

Job
Tracker

kerb(mapreduce)
Task
Tracker
job token

Task

HDFS
HDFS
HDFS...
MapReduce Task Security
• Users have separate task directories with
permissions set to 700.
• Distributed cache is now div...
Web UI
• MapReduce makes heavy use of Web UI for
displaying state of cluster and running jobs.
• HDFS also has a web brows...
Oozie

• Client authenticates to Oozie
– Custom auth for Yahoo!
• Oozie authenticates to HDFS and MapReduce as
“oozie” pri...
Proxy Services Trust Model
• Requires trust that service (eg. Oozie) principal is
secure.
• Explored and rejected
– Having...
Protocols
• RPC
– Change RPC to use SASL and either:
• Kerberos authentication (GSSAPI)
• Tokens (DIGEST-MD5)

– User’s Ke...
Protocols
• HTTP
– User/Browser facing
• Yahoo – Custom Authentication
• External – SPNEGO or Kerberos login module

– Web...
Summary
• RPC
– Kerberos
• Application to NameNode, JobTracker
• DataNode to NameNode
• TaskTracker to JobTracker

– Diges...
Task
Accessing as user
Accessing as user

Backyard

Browser, 2ndNN, fsck
Browser, 2ndNN, fsck

HFTP

HTTP

NN

DIGEST-MD5
...
Backyard

JT

HTTP

RPC

User
User

Kerberos

TT

Browser
Browser

TT

Browser
Browser

HTTP

RPC

Task

DIGEST-MD5

Backy...
Authentication Paths
Job
Tracker

Oozie

Browser

Name
Node

Data
Node

Task
Tracker

NFS
User
Process

Task

MObStor

Zoo...
Interfaces (and their scope and stability)
•

Imported Interfaces
–
–

SASL – Standard for supporting token and Kerberos a...
Pluggability
• Pluggability in Hadoop supports different environments
• HTTP browser user authentication
– Yahoo – Backyar...
Performance
• The authentication should not introduce
substantial performance penalties.
• Delegation token design to avoi...
Reliability and Availability
• The Kerberos KDC can not be a single point of failure.
– Kerberos clients automatically fai...
Operations and Monitoring
• The number of Kerberos authorizations will be
logged on the NameNode and JobTracker.
• Authori...
Upcoming SlideShare
Loading in...5
×

Hadoop Security Architecture

9,402

Published on

A deep dive on Hadoop security given at Yahoo to other architects when Yahoo was rolling out the first version of Hadoop with security 0.20.100.

Published in: Technology

Hadoop Security Architecture

  1. 1. Hadoop Security Architecture Owen O’Malley oom@yahoo-inc.com Hadoop
  2. 2. Outline • • • • • • Problem Statement Security Threats Solutions to Threats HDFS MapReduce Oozie • Interfaces • Performance • Reliability and Availability • Operations and Monitoring Hadoop 2
  3. 3. Problem Statement • The fundamental goal of adding Hadoop security is that Yahoo's data stored in HDFS must be secure from unauthorized access. Furthermore, it must do so without adding significant effort to operating or using the Grid. Based on that goal, there are a few implications. – All HDFS clients must be authenticated to ensure that the user is who they claim to be. That implies that all map/reduce users, including services such as Oozie, must be also authenticated and that tasks must run with the privileges and identity of the submitting user. – Since Data Nodes and TaskTrackers are entrusted with user data and credentials, they must authenticate themselves to ensure they are running as part of the Grid and are not trojan horses. – Kerberos will be the underlying authentication service so that users can be authenticated using their system credentials. Hadoop
  4. 4. Communication and Threats User Process Job Tracker Name Node Task Tracker Oozie Data Node Task NFS MObStor ZooKeeper Hadoop 4
  5. 5. Security Threats in Hadoop • User to Service Authentication – No User Authentication on NameNode or JobTracker • Client code supplies user and group names – No User Authorization on DataNode – Fixed in 0.21 • Users can read/write any block – No User Authorization on JobTracker • • • Users can modify or kill other user’s jobs Users can modify the persistent state of JobTracker Service to Service Authentication – No Authentication of DataNodes and TaskTrackers • • Users can start fake DataNodes and TaskTrackers No Encryption on Wire or Disk Hadoop
  6. 6. Solutions to Threats • Add Kerberos-based authentication to NameNode and JobTracker. • Add delegation tokens to HDFS and support for them in MapReduce. • Determine user’s group membership on the NameNode and JobTracker. • Protect MapReduce system directory from users. • Add authorization model to MapReduce so that only submitting user can modify or kill job. • Add Backyard authentication to Web UI’s. Hadoop 6
  7. 7. Out of Scope for 0.20.100 • Protecting against root on slave nodes: – Encryption of RPC messages – Encryption of block transfer protocol – Encryption of MapReduce transient files – Encryption of HDFS block files • Passing Kerberos tickets to MapReduce tasks for third party Kerborized services. Hadoop 7
  8. 8. HDFS Security • Users authenticate via Kerberos • MapReduce jobs can obtain delegation tokens for later use. • When clients are reading or writing an HDFS file, the NameNode generates a block access token that will be verified by the DataNode. Hadoop 8
  9. 9. HDFS Authentication ke r ) b(joe Application Name Node delg(jo e) MapReduce Task kerb(hdfs) bloc k to ken Data Node o ck t blo k en • Clients authenticate to NameNode via: – Kerberos – Delegation tokens • Client demonstrates authorization to DataNode via block access token • DataNode authenticates to NameNode via Kerberos Hadoop
  10. 10. What does this *really* look like? • Need a Kerberos ticket to work – kinit –l 7d oom@DS.CORP.YAHOO.COM – hadoop fs –ls – hadoop jar my.jar in-dir out-dir • Works using ticket cache! – Can display ticket cache with klist. Hadoop 10
  11. 11. Kerberos Dataflows Key Distribution Center Get TGT Request Service Ticket Name Node hdfs/host@YGRID User joe@DS.CORP Connect with Service Ticket Hadoop 11
  12. 12. Delegation Token • Advantages over using Kerberos directly: – Don’t trust JobTracker with credentials – Avoid MapReduce task authorization flood – Renewable by third party (ie. JobTracker) – Revocable when job finishes • tokenId = {owner prin, renewer prin, issueDate, maxDate} • tokenAuthenticator = HMAC(masterKey, tokenId) • Token = {tokenId, tokenAuthenticator} Hadoop 12
  13. 13. Block Access Token • Only NameNode knows the set of users allowed to access a specific block, so the NameNodes gives an authorized clients a block access token. • Capabilities include read, write, copy, or replace. • The NameNode and DataNodes share a dynamically rolled secret key to secure the tokens. • tokenId={expiration, keygen, owner, block, access} • tokenAuthenticator = HMAC(blockKey, tokenId) • token = {tokenId, tokenAuthenticator} Hadoop 13
  14. 14. MapReduce Security • Require Kerberos authentication from client. • Secure the information about pending and running jobs – Store the job configuration and input splits in HDFS under ~user/.staging/$jobid – Store the job’s location and secrets in private directory • JobTracker creates a random job token. It it used for: – Connecting to TaskTracker’s RPC – Authorizing http get for shuffle • HMAC(job token, URL) sent from reduce tasks to TaskTracker Hadoop 14
  15. 15. MapReduce Authentication Application kerb(joe) Job Tracker kerb(mapreduce) Task Tracker job token Task HDFS HDFS HDFS ) (joe elg d other credential trus t Other Service NFS • Client authenticates to JobTracker via Kerberos • TaskTracker authenticates to JobTracker via Kerberos • Task authenticates to the TaskTrackers using the job token • Task authenticates to HDFS using a delegation token • NFS is not Kerberized. Hadoop 15
  16. 16. MapReduce Task Security • Users have separate task directories with permissions set to 700. • Distributed cache is now divided based on the source’s visibility – Global – shared with other users – Private - protected from other users Hadoop 16
  17. 17. Web UI • MapReduce makes heavy use of Web UI for displaying state of cluster and running jobs. • HDFS also has a web browsing interface. • Use Backyard to authenticate Web UI users • Only allow submitting user of job to view stdout and stderr of job’s tasks. • HDFS web browser checks user’s authorization. Hadoop 17
  18. 18. Oozie • Client authenticates to Oozie – Custom auth for Yahoo! • Oozie authenticates to HDFS and MapReduce as “oozie” principal • “oozie” is configured as a super-user for HDFS and MapReduce and may act as other users. Hadoop 18
  19. 19. Proxy Services Trust Model • Requires trust that service (eg. Oozie) principal is secure. • Explored and rejected – Having user headless principals stored on Oozie machine “x/oozie” for user “x” – Passing user headless principal keytab to Oozie – Generalizing delegation token to have token granting tokens. Hadoop 19
  20. 20. Protocols • RPC – Change RPC to use SASL and either: • Kerberos authentication (GSSAPI) • Tokens (DIGEST-MD5) – User’s Kerberos tickets obtained at login used automatically. – Changes RPC format – Can easily add encryption later • Block transfer protocol – Block access tokens in data stream Hadoop 20
  21. 21. Protocols • HTTP – User/Browser facing • Yahoo – Custom Authentication • External – SPNEGO or Kerberos login module – Web Services • HFTP – Hadoop File Transfer Protocol • Others later • SPNEGO or Delegation Token via RPC – Shuffle • Use HMAC of URL hashed with Job Token Hadoop 21
  22. 22. Summary • RPC – Kerberos • Application to NameNode, JobTracker • DataNode to NameNode • TaskTracker to JobTracker – Digest-MD5 • MapReduce task to NameNode, TaskTracker • Block Access Token • Backyard – User to Web UI Hadoop 22
  23. 23. Task Accessing as user Accessing as user Backyard Browser, 2ndNN, fsck Browser, 2ndNN, fsck HFTP HTTP NN DIGEST-MD5 RPC Kerberos User (initial access), User (initial access), 2ndNN, Balancer 2ndNN, Balancer HTTP-DIGEST w/ delegation token HTTP-DIGEST w/ delegation token Task distcp accessing as user distcp accessing as user HFTP User, DN, Balancer, Task User, DN, Balancer, Task DN HTTP access token Task Socket Backyard Kerberos Browser Forward delegation token Forward delegation token Task distcp accessing as user distcp accessing as user Hadoop
  24. 24. Backyard JT HTTP RPC User User Kerberos TT Browser Browser TT Browser Browser HTTP RPC Task DIGEST-MD5 Backyard Kerberos -HMAC Local Task Local Task Hadoop Task Reduce Task getting Reduce Task getting Map output Map output
  25. 25. Authentication Paths Job Tracker Oozie Browser Name Node Data Node Task Tracker NFS User Process Task MObStor ZooKeeper Hadoop HTTP backyard HTTP HMAC RPC Kerberos RPC DIGEST Block Access Third Party 25
  26. 26. Interfaces (and their scope and stability) • Imported Interfaces – – SASL – Standard for supporting token and Kerberos authentication – GSSAPI – Kerberos part of SASL authentication – HMAC-SHA1 – Shared secret authentication – • JAAS – Java API for supporting authentication SPNEGO – Use Kerberos tickets over HTTP Exported Interfaces – Both Limited Private – – • HDFS adds a method to get delegation tokens RPC adds a doAs method Major Internal (Inter-system Interfaces) – MapReduce Shuffle uses HMAC-SHA1 – RPC uses Kerberos and DIGEST-MD5 Hadoop 26
  27. 27. Pluggability • Pluggability in Hadoop supports different environments • HTTP browser user authentication – Yahoo – Backyard – External – SPNEGO or Kerberos login module • RPC transport – SASL supports DIGEST-MD5, Kerberos, and others • Acquiring credentials – JAAS supports Kerberos, and others Hadoop 27
  28. 28. Performance • The authentication should not introduce substantial performance penalties. • Delegation token design to avoid authentication flood by MapReduce tasks • Required to be less than 3% on GridMix. Hadoop 28
  29. 29. Reliability and Availability • The Kerberos KDC can not be a single point of failure. – Kerberos clients automatically fail over to secondary KDC’s – Secondary KDC’s can be sync’ed automatically from the primary since the data rarely changes. • The cluster must remain stable when Kerberos fails. – The slaves (TaskTrackers and DataNodes) will lose their ability to reconnect to the master, when their RPC socket closes, their service ticket has expired, and both the primary and secondary KDC’s have failed. – Decided not to use special tokens to handle this case. • Once the MapReduce job is submitted, the KDC is not required for the job to continue running. Hadoop 29
  30. 30. Operations and Monitoring • The number of Kerberos authorizations will be logged on the NameNode and JobTracker. • Authorization failures will be logged. • Authentication failures will be logged. • The authorization logs will be a separate log4j logger, so they can be directed to a separate file. Hadoop 30
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×