Your SlideShare is downloading. ×
0
A TCloud Trender
徐啟超
WITHOUT KERBEROS
• Authorization
Ensuring the user can only do things that they are allowed to do
• Yes: Owner/Group Permi...
WITH KERBEROS
WITH KERBEROS
KERBEROS CONFIG BECOMING EASY
• Cloudera
• Cloudera Manager
• HDP
• Ambari – Security Wizard --- ambari-1.2.5
HADOOP GATEWAYS
client
Firewall
Gateway
Hadoop
Cluster
HADOOP GATEWAY - NOW
• Webhdfs
• Rest: curl "http://GATEWAYHOST/webhdfs/v1/PATH?[user.name=USER&]op=…”
• Hadoop: hadoop fs...
HADOOP GATEWAY - FUTURE
• Apache Knox Gateway
Provides a single point of authentication and access for Apache™ Hadoop® ser...
HADOOP GATEWAY - FUTURE
• Apache Knox Gateway
• Integrate with the existing frameworks for Active Directory /LDAP
• Shell ...
HADOOP DATA ENCRYPTION
• Disk Encryption
• Partition Encryption  dm-crypt
• File System Encryption
• Folder Encryption  ...
HADOOP ENCRYPTION FRAMEWORK - API
Local
File
HDFS
encrypt/
decrypt
HDFS
File
Encrypt/decry
pt
HADOOP ENCRYPTION FRAMEWORK - MR
File Map File Reduce
HDFS
HDFS
File
Encryption/Decryption All the Path(Stages)
JIRAS
• hadoop-9331: Hadoop crypto codec framework and crypto codec implementations
• hadoop-9332: Crypto codec implementa...
Brief
• Two Crypto Typical Case in Hadoop
• Crypto API Case: Using AES Key (Store in KeyStore) to Encrypt/Decrypt Data
• M...
KEY STORE TOOL - KEYTOOL
A key and certificate management utility.
• Create & Store an AES key
• keytool -keystore /tmp/hb...
CRYPTO API CASE:
USING AES KEY (STORE IN KEYSTORE) TO
ENCRYPT/DECRYPT DATA
CRYPTO API CASE: USING AES KEY (STORE IN
KEYSTORE) TO ENCRYPT/DECRYPT DATA
Use Crypto API to retrieve AES secret key from ...
KeyProvider - KeyStoreKeyProvider
• To retrieve secret key from specified Key Store File
• Parameters
• keyStoreUrl & pass...
CryptoContext
• To store key related info
• Key Attributes
• Raw Key Data
• Key Type: SYMMETRIC_KEY, PUBLIC_KEY, PRIVATE_K...
CryptoCodec
• A wrap, contain CryptoContext and provide Crypto IO Stream
• Major member
• CryptoContext
• Crypto IO Stream...
SAMPLE CODE --- FILE ENCRYPTION
SAMPLE CODE --- FILE ENCRYPTION - conti
MR CRYPTOCONTEXT CASE:
ENCRYPT THE MR OUTPUT
MR CRYPTOCONTEXT CASE: ENCRYPT THE MR
OUTPUT
Using provided CryptoContext to encrypt the Map Reduce output
• KeyProviderCo...
KeyProviderConfig
• Members
• keyProvider
• keyProviderParameters
• Methods
• getKeyProvider()
• getKeyProviderParameters()
CryptoContextProvider
Provide several static helper methods to update Crypto related Job Configurations. For
example, stor...
AbstractCryptoContextProvider
 Methods
 setInputCryptoContextProvider (jobConf, cryptoContextProviderClass, secrets,
par...
FileMatchCryptoContextProvider
Provides the ability to select the appropriate CryptoContext according to the file path
Fil...
KeyProviderCryptoContextProvider
Not only include the capability of FileMatchCryptoContextProvider also provide the abilit...
SAMPLE CODE - ENCRYPT THE MR OUTPUT
Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
JobCon...
SAMPLE CODE - ENCRYPT THE MR OUTPUT
Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
JobCon...
SAMPLE CODE - ENCRYPT THE MR OUTPUT
- Conti
FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00",
Key.KeyTyp...
SAMPLE CODE - ENCRYPT THE MR OUTPUT
- Conti
FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00",
Key.KeyTyp...
SAMPLE CODE - ENCRYPT THE MR OUTPUT
- Conti
FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00",
Key.KeyTyp...
MORE IN KeyProviderCryptoContextProvider
• Using asymmetric key (RSA) to protect Parameters & Secrets
MORE IN KeyProviderCryptoContextProvider
• Using asymmetric key (RSA) to protect Parameters & Secrets
CredentialProtection...
MORE IN KeyProviderCryptoContextProvider - Conti
• How to use Customized KeyProvider in KeyProviderCryptoContextProvider
S...
TOOL – DISTCRYPTO
Use MapReduce Job to encrypt, decrypt or key rotate multiple files
• Usage
• -op <operation> : "encrypt”...
TOOL – DISTCRYPTO - conti
• Source Definition File (XML format)
• src
• path
• format:
• raw
• Sequence
• the full class n...
TOOL – DISTCRYPTO - conti
• Encryption Sample
• command
• hadoop distcrypto -op encrypt -ek
21EF7D7487F69A19E552C1274A9FCA...
HBASE ENCRYPTION
HBASE-7544
HBASE ENCRYPTION – HBASE-7544
• Introduce transparent encryption of HBase on disk data.
• Transparent encryption at the CF...
HBASE ENCRYPTION – HBASE-7544
HBASE ENCRYPTION – HBASE-7544
HFile
Block0
……
Block N
Meta Block0
……
Meta Block N
File Info
Data Block Index
Mwta Block In...
HBASE-7544 SETTINGS
1. Set up the keystore with a secret key
Create a secret key of appropriate length for AES.
$ keytool ...
HBASE-7544 SETTINGS
2. Configure HBase to use the keystore
Add this to the hbase-site.xml file:
<property>
<name>hbase.cry...
HBASE-7544 SETTINGS
3. Create the table
$ ./bin/hbase shell
hbase(main):001:0> create 'test', {NAME=>'t', CRYPTO=>'AES',
C...
HBASE-7544
• CF key rotation
• CF key is changed by modifying the column descriptor via
HBaseAdmin.
• Then, major compacti...
OTHER RELATED JIRAS
• MAPREDUCE-4491: Encryption and Key Protection
• 4550: Key Protection : Define Encryption and Key Pro...
SECURITY WEB KEYSTORE SERVER
safe (http://benoyantony.github.com/safe/)
Web service based keystore
Support ACL Per Key
Aut...
OTHER TODOs
• Hive support
• https://issues.apache.org/jira/browse/HIVE-5207
• Support data encryption for Hive tables
• h...
Q & A
Hadoop Security Now and Future
Hadoop Security Now and Future
Hadoop Security Now and Future
Hadoop Security Now and Future
Upcoming SlideShare
Loading in...5
×

Hadoop Security Now and Future

1,260

Published on

In Hadoop in Taiwan 2013 event, engineer of TCloud Computing presented the security concepts and features of Hadoop, how to script Crypto API, configuration details and future development.

Published in: Technology, Education
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,260
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
73
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Transcript of "Hadoop Security Now and Future"

  1. 1. A TCloud Trender 徐啟超
  2. 2. WITHOUT KERBEROS • Authorization Ensuring the user can only do things that they are allowed to do • Yes: Owner/Group Permission • Authentication Ensuring the user is who they claim to be • NO
  3. 3. WITH KERBEROS
  4. 4. WITH KERBEROS
  5. 5. KERBEROS CONFIG BECOMING EASY • Cloudera • Cloudera Manager • HDP • Ambari – Security Wizard --- ambari-1.2.5
  6. 6. HADOOP GATEWAYS client Firewall Gateway Hadoop Cluster
  7. 7. HADOOP GATEWAY - NOW • Webhdfs • Rest: curl "http://GATEWAYHOST/webhdfs/v1/PATH?[user.name=USER&]op=…” • Hadoop: hadoop fs -fs webhdfs://GATEWAYHOST:14000 -cat FILe_PATH • Oozie • REST API , supports direct submission of MapReduce, Pig, and Hive jobs • Steps • Use webhdfs to upload your files and jars • create an oozie workflow • Hbase • Hbase Stargate Rest Gateway • Hbase Thrift server
  8. 8. HADOOP GATEWAY - FUTURE • Apache Knox Gateway Provides a single point of authentication and access for Apache™ Hadoop® services in a cluster
  9. 9. HADOOP GATEWAY - FUTURE • Apache Knox Gateway • Integrate with the existing frameworks for Active Directory /LDAP • Shell and Rest Interface support • Currently working on kerberized cluster support
  10. 10. HADOOP DATA ENCRYPTION • Disk Encryption • Partition Encryption  dm-crypt • File System Encryption • Folder Encryption  encryptfs • Hadoop Encryption Framework • Just encrypt what it should be
  11. 11. HADOOP ENCRYPTION FRAMEWORK - API Local File HDFS encrypt/ decrypt HDFS File Encrypt/decry pt
  12. 12. HADOOP ENCRYPTION FRAMEWORK - MR File Map File Reduce HDFS HDFS File Encryption/Decryption All the Path(Stages)
  13. 13. JIRAS • hadoop-9331: Hadoop crypto codec framework and crypto codec implementations • hadoop-9332: Crypto codec implementations for AES • hadoop-9333: Hadoop crypto codec framework based on compression codec • mapreduce-5025: Key Distribution and Management for supporting crypto codec in Map Reduce • hbase-7544: Transparent table/CF encryption
  14. 14. Brief • Two Crypto Typical Case in Hadoop • Crypto API Case: Using AES Key (Store in KeyStore) to Encrypt/Decrypt Data • MR CryptoContext Case: Encrypt the MR output • Tool – Distcrypto • Hbase Encryption • Other Related JIRAs and Security Key Store(Manager) • TODOs
  15. 15. KEY STORE TOOL - KEYTOOL A key and certificate management utility. • Create & Store an AES key • keytool -keystore /tmp/hbase.jks -storetype jceks -storepass 123456 -genseckey - keyalg AES -keysize 256 -alias hbase • Create & Store an RSA Private Key • keytool -genkey -keyalg RSA -keysize 2048 -storetype jceks -storepass 123456 - keystore privateKeyStore.jks -alias testPrivate • Export Certificate from KeyStore to a cert file • keytool -export -keystore privateKeyStore.jks -storetype jceks -storepass 123456 - alias testPrivate -file publicKey.crt • Import a cert file to a KeyStore • keytool -import -trustcacerts -file publicKey.crt -storetype jceks -storepass 123456 - alias testPublic -keystore publicKeyStore.jks
  16. 16. CRYPTO API CASE: USING AES KEY (STORE IN KEYSTORE) TO ENCRYPT/DECRYPT DATA
  17. 17. CRYPTO API CASE: USING AES KEY (STORE IN KEYSTORE) TO ENCRYPT/DECRYPT DATA Use Crypto API to retrieve AES secret key from a key store file and use the key to encrypt/decrypt data • KeyProvider • CryptoContext • CryptoCodec • Sample Code
  18. 18. KeyProvider - KeyStoreKeyProvider • To retrieve secret key from specified Key Store File • Parameters • keyStoreUrl & password • keyStoreType: JCE, JCEKS …… • keyPasswordFile & sharePassword • Initial: keyProvider.init(String parameters) • String parameters = “file:///etc/keystore/mapred.jks&keyStoreType=JCEKS &password=123456” • String parameters = KeyStoreKeyProvider.getKeyStoreParameterString(fileUrl, StoreType, StorePassword, null, true); • Get: keyProvider.getKeys(String [])
  19. 19. CryptoContext • To store key related info • Key Attributes • Raw Key Data • Key Type: SYMMETRIC_KEY, PUBLIC_KEY, PRIVATE_KEY, CERTIFICATE • Cryptographic Algorithm: e.g AES • Cryptographic Length
  20. 20. CryptoCodec • A wrap, contain CryptoContext and provide Crypto IO Stream • Major member • CryptoContext • Crypto IO Stream Method • createOutputStream(……) • createInputStream(……)
  21. 21. SAMPLE CODE --- FILE ENCRYPTION
  22. 22. SAMPLE CODE --- FILE ENCRYPTION - conti
  23. 23. MR CRYPTOCONTEXT CASE: ENCRYPT THE MR OUTPUT
  24. 24. MR CRYPTOCONTEXT CASE: ENCRYPT THE MR OUTPUT Using provided CryptoContext to encrypt the Map Reduce output • KeyProviderConfig • CryptoContextProvider • Sample Code
  25. 25. KeyProviderConfig • Members • keyProvider • keyProviderParameters • Methods • getKeyProvider() • getKeyProviderParameters()
  26. 26. CryptoContextProvider Provide several static helper methods to update Crypto related Job Configurations. For example, store the following Parameters and Secrets to the Job Credential in the secret key list • mapred.[[[STAGE]]].crypto.context.provider.parameters • mapred.[[[STAGE]]].crypto.context.secrets [[[STAGE]]]: input, output, map.output  AbstractCryptoContextProvider  FileMatchCryptoContextProvider  KeyProviderCryptoContextProvider Credentials credentials = jobConf.getCredentials(); credentials.addSecretKey(new Text("mapred.map.output.crypto.context.provider.parameters"), parameters); credentials.addSecretKey(new Text("mapred.map.output.crypto.context.secrets"), secrets);
  27. 27. AbstractCryptoContextProvider  Methods  setInputCryptoContextProvider (jobConf, cryptoContextProviderClass, secrets, parameters)  setMapOutputCryptoContextProvider (jobConf, cryptoContextProviderClass, secrets, parameters)  setOutputCryptoContextProvider (jobConf, cryptoContextProviderClass, secrets, parameters)
  28. 28. FileMatchCryptoContextProvider Provides the ability to select the appropriate CryptoContext according to the file path FileMatches fileMatches = new FileMatches(KeyContext.derive("12345678")); fileMatches.addMatch("^.*/input1.intel_aes$", KeyContext.derive("1234")); fileMatches.addMatch("^.*/input2.intel_aes$", KeyContext.derive("5678")); FileMatchCryptoContextProvider.setInputCryptoContextProvider(jobConf, fileMatches, null);
  29. 29. KeyProviderCryptoContextProvider Not only include the capability of FileMatchCryptoContextProvider also provide the ability to retrieve the Key from Key Store FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00", Key.KeyType.SYMMETRIC_KEY, "AES", 128)); String keyStoreFile = "file:///" + KEYSTORE_HOME + "/mr.jks"; String keyStorePassword = "12345678"; KeyProviderConfig keyProviderConfig = KeyProviderCryptoContextProvider.getKeyStoreKeyProviderConfig( keyStoreFile, "JCEKS", keyStorePassword, null, true); KeyProviderCryptoContextProvider.setInputCryptoContextProvider(jobConf, fileMatches, true, keyProviderConfig);
  30. 30. SAMPLE CODE - ENCRYPT THE MR OUTPUT Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); JobConf jobConf = (JobConf)job.getConfiguration();
  31. 31. SAMPLE CODE - ENCRYPT THE MR OUTPUT Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); JobConf jobConf = (JobConf)job.getConfiguration(); FileOutputFormat.setOutputCompressorClass(job, AESCodec.class); jobConf.set(AESCodec.CRYPTO_COMPRESSOR, org.apache.hadoop.io.compress.SnappyCodec);
  32. 32. SAMPLE CODE - ENCRYPT THE MR OUTPUT - Conti FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00", Key.KeyType.SYMMETRIC_KEY, "AES", 256));
  33. 33. SAMPLE CODE - ENCRYPT THE MR OUTPUT - Conti FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00", Key.KeyType.SYMMETRIC_KEY, "AES", 256)); String keyStoreFile = "file:///" + KEYSTORE_HOME + "/mr.jks"; String keyStorePassword = "12345678"; KeyProviderConfig keyProviderConfig = KeyProviderCryptoContextProvider.getKeyStoreKeyProviderConfig( keyStoreFile, "JCEKS", keyStorePassword, null, true);
  34. 34. SAMPLE CODE - ENCRYPT THE MR OUTPUT - Conti FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00", Key.KeyType.SYMMETRIC_KEY, "AES", 256)); String keyStoreFile = "file:///" + KEYSTORE_HOME + "/mr.jks"; String keyStorePassword = "12345678"; KeyProviderConfig keyProviderConfig = KeyProviderCryptoContextProvider.getKeyStoreKeyProviderConfig( keyStoreFile, "JCEKS", keyStorePassword, null, true); KeyProviderCryptoContextProvider.setOutputCryptoContextProvider(jobConf, fileMatches, false, keyProviderConfig); ……. job.waitForCompletion(true);
  35. 35. MORE IN KeyProviderCryptoContextProvider • Using asymmetric key (RSA) to protect Parameters & Secrets
  36. 36. MORE IN KeyProviderCryptoContextProvider • Using asymmetric key (RSA) to protect Parameters & Secrets CredentialProtection credentialProtection = new CredentialProtection(jobConf, RSACredentialProtectionCodec.class, encryptionKeyProviderConfig, encryptionKeyName, decryptionKeyProviderConfig, decryptionKeyName); KeyProviderCryptoContextProvider.setInputCryptoContextProvider( jobConf, fileMatches, false, keyProviderConfig, credentialProtection);
  37. 37. MORE IN KeyProviderCryptoContextProvider - Conti • How to use Customized KeyProvider in KeyProviderCryptoContextProvider String keyProviderParameters = KeyStoreKeyProvider.getKeyStoreParameterString( keyStoreFile, keyStoreType, keyStorePassword, keyStorePasswordFile, sharedPassword); KeyProviderConfig keyProviderConfig = new KeyProviderConfig( CustomizeKeyStoreKeyProvider.class.getName(), keyProviderParameters);
  38. 38. TOOL – DISTCRYPTO Use MapReduce Job to encrypt, decrypt or key rotate multiple files • Usage • -op <operation> : "encrypt”, "decrypt" and "rotate” • --ek <encryption key> • -dk <decryption key> • -src <source definition file> • -dst <dest url> • -log <log url>
  39. 39. TOOL – DISTCRYPTO - conti • Source Definition File (XML format) • src • path • format: • raw • Sequence • the full class name of a class which implement CryptoHandler for customized format. • includeFilter & excludeFilter • stripSuffix & appendSuffix • keyClassName & valueClassName.
  40. 40. TOOL – DISTCRYPTO - conti • Encryption Sample • command • hadoop distcrypto -op encrypt -ek 21EF7D7487F69A19E552C1274A9FCAC721EF7D7487F69A19E552C1274A9F CAC7 -log /tmp/log.distcrypto.encrypt -src file:///working/crypto_encrypt.xml • Source Definition File (crypto_encrypt.xml) • TODO: Not support retrieve keys from key store --- Not Good <configuration><src> <path>/tmp/install.log</path> <format>raw</format> <appendSuffix>.encrypted</appendSuffix> </src></configuration>
  41. 41. HBASE ENCRYPTION HBASE-7544
  42. 42. HBASE ENCRYPTION – HBASE-7544 • Introduce transparent encryption of HBase on disk data. • Transparent encryption at the CF level • Two-tier key architecture for consistency with best practices for this feature in the RDBMS world • Flexible and non-intrusive key rotation
  43. 43. HBASE ENCRYPTION – HBASE-7544
  44. 44. HBASE ENCRYPTION – HBASE-7544 HFile Block0 …… Block N Meta Block0 …… Meta Block N File Info Data Block Index Mwta Block Index Fixed File Trailer Key block data format 1 byte ordinal 4 bytes key data length encrypted key data Encryption KeyBlock Offset
  45. 45. HBASE-7544 SETTINGS 1. Set up the keystore with a secret key Create a secret key of appropriate length for AES. $ keytool -keystore /path/to/hbase/conf/hbase.jks -storetype jceks -storepass password -genseckey -keyalg AES -keysize 256 -alias ${USER} Press RETURN to store the key with the same password as the store
  46. 46. HBASE-7544 SETTINGS 2. Configure HBase to use the keystore Add this to the hbase-site.xml file: <property> <name>hbase.crypto.keyprovider</name> <value>org.apache.hadoop.io.crypto.KeyStoreKeyProvider</value> </property> <property> <name>hbase.crypto.keyprovider.parameters</name> <value><![CDATA[keyStoreUrl=file:///path/to/hbase/conf/ hbase.jks&keyStoreType=JCEKS&password=password]]></value> </property>
  47. 47. HBASE-7544 SETTINGS 3. Create the table $ ./bin/hbase shell hbase(main):001:0> create 'test', {NAME=>'t', CRYPTO=>'AES', CRYPTO_KEY=>'123456'}
  48. 48. HBASE-7544 • CF key rotation • CF key is changed by modifying the column descriptor via HBaseAdmin. • Then, major compaction is triggered either on the table at once or region by region. • Performance • Using this AES-NI codec, HFile read and write code paths introduces an overhead roughly on par with GZIP compression for reads, and half that as for writes.
  49. 49. OTHER RELATED JIRAS • MAPREDUCE-4491: Encryption and Key Protection • 4550: Key Protection : Define Encryption and Key Protection interfaces and default implementation • 4551: Key Protection : Add ability to read keys and protect keys in JobClient and TTS/NodeManagers • 4552: Encryption: Add support for PGP Encryption • 4553: Key Protection : Implement KeyProvider to read key from a WebService Based KeyStore • 5025: Key Distribution and Management for supporting crypto codec in Map Reduce
  50. 50. SECURITY WEB KEYSTORE SERVER safe (http://benoyantony.github.com/safe/) Web service based keystore Support ACL Per Key Authenticates the user using SPNego Base on Cloudera Alfredo, a Java library consisting of a client and a server components to enable Kerberos SPNEGO authentication for HTTP. WEB Server (safe(alfredo)) KDC user authorization authentication MR/Hbase + WebStoreKeyProvider
  51. 51. OTHER TODOs • Hive support • https://issues.apache.org/jira/browse/HIVE-5207 • Support data encryption for Hive tables • https://issues.apache.org/jira/browse/HIVE-4227 • Add column level encryption to ORC files (Created: 25/Mar/13 17:14) • Pig support • https://issues.apache.org/jira/browse/PIG-3289 • Encryption aware load and store functions
  52. 52. Q & A
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×