Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ORC Column Encryption

219 views

Published on

To provide better security, ORC files are adding column encryption. Column encryption provides the ability to grant access to different columns within the same file. All of the encryption is handled transparently to the user.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

ORC Column Encryption

  1. 1. © Hortonworks Inc. 2018 ORC Column Encryption May 2018 Page 1 Owen O’Malley owen@hortonworks.com @owen_omalley
  2. 2. © Hortonworks Inc. 2018 Controlling Sensitive Data Page 2
  3. 3. © Hortonworks Inc. 2018 What is the Problem? Page 3
  4. 4. © Hortonworks Inc. 2018 Partial Solution – HDFS Encryption Page 4
  5. 5. © Hortonworks Inc. 2018 KeyProvider API Page 5
  6. 6. © Hortonworks Inc. 2018 HDFS Encryption Limitations Page 6
  7. 7. © Hortonworks Inc. 2018 Partial Solution – Hive Server 2 Page 7
  8. 8. © Hortonworks Inc. 2018 Hive Architecture with Hive Server 2 Page 8
  9. 9. © Hortonworks Inc. 2018 Hive Server 2 Limitations Page 9
  10. 10. © Hortonworks Inc. 2018 Partial Solution – Encryption UDF Page 10
  11. 11. © Hortonworks Inc. 2018 Partial Solution – Separate tables Page 11
  12. 12. © Hortonworks Inc. 2018 Solution – Columnar Encryption Page 12
  13. 13. © Hortonworks Inc. 2018 Key Management Page 13
  14. 14. © Hortonworks Inc. 2018 Attribute-Based Access Control (ABAC) Page 14
  15. 15. © Hortonworks Inc. 2018 Key Disposal Page 15
  16. 16. © Hortonworks Inc. 2018 Encrypting Columns Page 16
  17. 17. © Hortonworks Inc. 2018 Data Masking Page 17
  18. 18. © Hortonworks Inc. 2018 Data Masking Page 18
  19. 19. © Hortonworks Inc. 2018 ORC File Format Page 19 File Footer Postscript Index Data Row Data Stripe Footer ~200MBStripe Index Data Row Data Stripe Footer ~200MBStripe Index Data Row Data Stripe Footer ~200MBStripe Column 1 Column 2 Column 7 Column 8 Column 3 Column 6 Column 4 Column 5 Column 1 Column 2 Column 7 Column 8 Column 3 Column 6 Column 4 Column 5 Stream 2.1 Stream 2.2 Stream 2.3 Stream 2.4
  20. 20. © Hortonworks Inc. 2018 ORC Column Encryption Page 20
  21. 21. © Hortonworks Inc. 2018 ORC Write Pipeline Page 21
  22. 22. © Hortonworks Inc. 2018 Encryption flow Page 22
  23. 23. © Hortonworks Inc. 2018 AES/CTR Page 23
  24. 24. © Hortonworks Inc. 2018 Conclusions Page 24
  25. 25. © Hortonworks Inc. 2018 Challenges Page 25
  26. 26. © Hortonworks Inc. 2018 Limitations Page 26
  27. 27. © Hortonworks Inc. 2018 Questions? Page 27

×