Fine-grained data protection at a column level in data lake environments has become a mandatory requirement to demonstrate compliance with multiple local and international regulations across many industries today. ORC is a self-describing type-aware columnar file format designed for Hadoop workloads that provides optimized streaming reads, but with integrated support for finding required rows quickly. In this talk, we will outline the progress made in Apache community for adding fine-grained column level encryption natively into ORC format that will also provide capabilities to mask or redact data on write while protecting sensitive column metadata such as statistics to avoid information leakage. The column encryption capabilities will be fully compatible with Hadoop Key Management Server (KMS) and use the KMS to manage master keys providing the additional flexibility to use and manage keys per column centrally. An end to end scenario that demonstrates how this capability can be leveraged will be also demonstrated.

1. Fine Grained Access Control for Big
Data: ORC Column Encryption
Owen O’Malley
owen@cloudera.com
@owen_omalley
March 2019
Srikanth Venkat
svenkat@cloudera.com
@srikvenk

2. 2 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Who Are We?
• Owen
• Worked on Hadoop since Jan 2006
• MapReduce, Security, Hive, and ORC
• Founder & Technical Fellow
• Srikanth
• Senior Director, Product Management (Security &
Governance portfolio)
• Apache Ranger, Apache Knox, Apache Atlas, ODPi
• Security, Data Stewardship, Metadata, Governance areas

3. 3 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Security & Data Protection in Hadoop

4. 4 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Example Data Lake Scenario
Marketing
Demographics
Electronic
medical records
CRM
POS
(Structured)(Structured) (Structured) (Structured) (Structured)
Cluster 1: Dublin Cluster 2: San Francisco
(Unstructured)(Unstructured)(Unstructured)
Cluster 3: Prague
(Structured)
On Premise Data Lakes
(Unstructured)(Structured) (Unstructured) (Structured)
Cloud Data Lakes
Social
Weblogs & Feeds
Transactional
Mobile
IoT
Personal Data

5. 5 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
What’s different about the Big Data context?
à Breaking down silos: fantastic for analytics, but leads to increased security
challenges
– Centralized data lake with multi-tenancy requires secure (and easy) authentication and fine-
grained authorization
à Data democratization and the Data Scientist role (often a data superuser
with elevated privileges)
à Data is maintained over a long duration
à Cloud and Hybrid architectures spanning data center and (multiple) public
clouds further broaden the attack surface area and present novel
authentication and authorization challenges
à Along with adherence to security fundamentals and defense in-depth, a
data-centric approach to security becomes critical

6. 6 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Watch Towers
Limited Entry Points
Moat
Kerberos
Securing your data lake
High Hard Walls
Check Identity
Inner Walls
Firewall
Encryption, TLS, Key
Trustee, Navigator
Encrypt, Ranger KMS
LDAP/AD
Apache Knox: AuthN, API
Gateway, Proxy, SSO
Apache Ranger : ABAC
AuthZ, Audits,
Anonymization
Apache Sentry: RBAC
AuthZ

7. 7 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Data Protection in Hadoop
must be applied at three different layers
in Apache Hadoop
Storage: encrypt data while it is at rest
Transparent Data Encryption in HDFS, Navigator Key Trustee, Navigator
Encrypt, Ranger KMS + HSM, Partner Products (HPE Voltage, Protegrity,
Dataguise)
Transmission: encrypt data as it is in motion
Wire encryption (TLS, SASL,..)
Upon Access: apply restrictions when accessed
Apache Ranger (Dynamic Column Masking + Row Filtering), Partner
Masking + Encryption
Data Protection

8. 8 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Encryption of Data in Hadoop
Volume
Encryption
Protects data after physical theft
or accidental loss of a disk volume.
Entire volume is encrypted: very
coarse-grained security
Does not protect against viruses or
other attacks that occur while a
system is running.
Application-
level encryption
Encryption within an application
running on top of Hadoop
Supports a higher level of
granularity and prevents "rogue
admin" access
Adds a layer of complexity to the
application architecture.
HDFS data-at-
rest encryption
Encrypts selected files and
directories stored ("at rest") in
HDFS.
Uses specially designated HDFS
directories known as "encryption
zones.”
End-to-end encryption of data
read from and written to HDFS.
HDFS does not have access to
unencrypted data or keys.

9. 9 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Dynamic Row Filtering & Column Masking With Apache Ranger & Apache Hive
User 2: Ivanna
Location : EU
Group: HR
User 1: Joe
Location : US
Group: Analyst
Original Query:
SELECT country, nationalid,
ccnumber, mrn, name FROM
ww_customers
Country National ID CC No DOB MRN Name Policy ID
US 232323233 4539067047629850 9/12/1969 8233054331 John Doe nj23j424
US 333287465 5391304868205600 8/13/1979 3736885376 Jane Doe cadsd984
Germany T22000129 4532786256545550 3/5/1963 876452830A Ernie Schwarz KK-2345909
Country National ID CC No MRN Name
US xxxxx3233 4539 xxxx xxxx xxxx null John Doe
US xxxxx7465 5391 xxxx xxxx xxxx null Jane Doe
Ranger Policy Enforcement
Query Rewritten based on Dynamic Ranger Policies:
Filter rows by region & apply relevant column masking
Users from US Analyst group see data for US
persons with CC and National ID (SSN) as
masked values and MRN is nullified
Country National ID Name MRN
Germany T22000129 Ernie Schwarz 876452830A
EU HR Policy Admins can see
unmasked but are restricted by row
filtering policies to see data for EU
persons only
Original Query:
SELECT country, nationalid,
name, mrn FROM
ww_customers
Analysts
HR Marketing

10. 10 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Framing the Problem…..
• Related data, different security requirements
• Authorization – who can see it
• Audit – track who read it
• Encrypt on disk – regulatory
• File-level (or blob) granularity isn’t enough
• File systems don’t understand columns

11. 11 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Requirements
• Readers should transparently decrypt data
• If and only if the user has access to the key
• The data must be decrypted locally
• Columns are only decrypted as necessary
• Master keys must be managed securely
• Support for Key Management Server & hardware
• Support for key rolling

12. 12 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Partial Solutions

13. 13 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Partial Solution – HDFS Encryption
• Transparent HDFS Encryption
• Encryption zones
• HDFS directory trees
• Unique master key for each zone
• Client decrypts data
• Key Management via KeyProvider API

14. 14 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
HDFS Encryption Limitations
• Very coarse protection
• Only entire directory subtrees
• No ability to protect columns
• A lot of users need access to keys
• Moves between zones is painful
• When writing with Hive, data is moved
multiple times per a query

15. 15 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Hive Server 2 Limitations
• Limits access to Hive SQL
• Only user ‘hive’ has access
• Breaks Hadoop’s multi-paradigm data access
• Many customers use both Hive & Spark
• JDBC is not distributed
• New Spark to LLAP connector addresses this

16. 16 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Partial Solution – Separate tables
• Split private information out of tables
• Separate directories in HDFS
• HDFS and/or HS2 authorization
• Enables HDFS encryption
• Limitations
• Need to join with other tables
• Higher operational overhead

17. 17 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Partial Solution – Encryption UDF
• Hive has user defined functions
• aes_encrypt and aes_decrypt
• Limitations
• Key management is problematic
• Encryption is not seeded
• Size of value leaks information

18. 18 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Solution

19. 19 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Columnar Encryption
• Columnar file formats (eg. ORC)
• Write data in columns
• Column projection
• Better compression
• Encryption works really well
• Only encrypt bytes for column
• Can store multiple variants of data

20. 20 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
ORC File Format
File Footer
Postscript
Index Data
Row Data
Stripe Footer
~200MBStripe
Index Data
Row Data
Stripe Footer
~200MBStripe
Index Data
Row Data
Stripe Footer
~200MBStripe Column 1
Column 2
Column 7
Column 8
Column 3
Column 6
Column 4
Column 5
Column 1
Column 2
Column 7
Column 8
Column 3
Column 6
Column 4
Column 5
Stream 2.1
Stream 2.2
Stream 2.3
Stream 2.4

21. 21 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
User Experience
• Set table properties for encryption
• orc.encrypt.pii = ”ssn,email”
• orc.encrypt.credit = “card_info”
• Define where to get the encryption keys
• Configuration defines the key provider via URI

22. 22 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Key Management
• Create a master key for each use
• “pii”, “pci”, or “hipaa”
• Each column in each file uses unique local key
• Allows audit of which users read which files
• Ranger policies limit access to keys
• Who, What, When, Where

23. 23 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
KeyProvider API
• Provides limited access to encryption keys
• Encrypts or decrypts local keys
• Users are never given master keys
• Key versions and key rolling of master keys
• Allows 3rd party plugins
• Supports Cloud, Hadoop or Ranger KMS

24. 24 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Encryption Data Flow

25. 25 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Encryption Flow
• Local key
• Random for each encrypted column in file
• Encrypted w/ master key by KMS
• Encrypted local key is stored in file metadata
• IV is generated to be unique
• Column, kind, stripe, & counter

26. 26 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Static Data Masking
• What happens without key access?
• Define static masks
• Nullify – all values become null
• Redact – mask values ‘Xxxxx Xxxxx!’
• Can define ranges to unmask
• SHA256 – replace with SHA256
• Custom - user defined

27. 27 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Data Masking
• Anonymization is hard!
• AOL search logs
• Netflix prize datasets
• NYC taxi dataset
• Always evaluate security tradeoffs
• Tokenization is a useful technique
• Assign arbitrary replacements

28. 28 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Key Disposal
• Often need to keep data for 90 days
• Currently the data is written twice
• With column encryption:
• Roll keys daily
• Delete master key after 90 days

29. 29 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
ORC Encryption Design
• Write both variants of streams
• Masked unencrypted
• Unmasked encrypted
• Encrypt both data and statistics
• Maintain compatibility for old readers
• Read unencrypted variant
• Preserve ability to seek in file

30. 30 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
ORC Write Pipeline
• Streams go through pipeline
• Run length encoding
• Compression (zlib, snappy, or lzo)
• Encryption
• Encryption is AES/CTR
• Allows seek
• No padding

31. 31 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Conclusions

32. 32 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Conclusions
• ORC column encryptions provides
• Transparent encryption
• Multi-paradigm column security
• Audit logging (via KMS logging)
• Static masking
• Supports file merging
• Different stripes with different local key

33. 33 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Integration with Other Tools
• Hive & Spark
• No change other than defining table properties
• Apache Hive’s LLAP
• Cache and fast processing of SQL queries
• Column encryption changes internal interfaces
• Cache both encrypted and unencrypted variants
• Ensure audit log reflects end-user and what they accessed

34. 34 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Limitations
• Need encryption policy for write
• Current Atlas & Ranger tags lag data
• Auto-discovery requires pre-access
• Changes to masking policy
• Need to re-write files
• Need additional data masks
• Credit card, addresses, etc.
• Decrypted local keys could be saved

35. 35 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Thank you!
Twitter: @owen_omalley
Email: owen@cloudera.com