Chlorine is a tool that scans Hadoop clusters to discover and protect sensitive data. It detects personally identifiable information like emails and credit cards stored in HDFS files using regular expressions. Chlorine analyzes the results to identify sensitive data, which can then be restricted, encrypted, or masked for protection. It supports Hadoop distributions, Hive, Kerberos authentication, and auditing while controlling access to scans and results.

1. DataApps Corporation 1
Discover Sensitive Data In Hadoop
using
Chlorine
Benoy Antony

2. DataApps Corporation 2
About me
●
Focus on securing Hadoop clusters
●
Apache Hadoop Committer
–
Contributed features to security and HDFS
●
Founder of DataApps - http://dataApps.io

3. DataApps Corporation 3
Problem
●
Users store many kinds of data in HDFS
●
There could be sensitive data stored
without sufficient protection
●
Results in non-compliance

4. DataApps Corporation 4
Solution
●
Scan the data against patterns matching
different items like email, credit card
●
Analyze the results for accuracy
●
If sensitive data identified, take action

5. DataApps Corporation 5
Protecting sensitive Data
–
Restrict access to the data using permissions
and extended ACLS
–
Encrypt the data
●
HDFS Encryption for HDFS files
●
Column level encryption for Hive columns
–
Mask the values

6. DataApps Corporation 6
Introducing Chlorine
●
Scans HDFS directories against patterns
to detect sensitive data elements.
●
Reports sensitive elements detected.

7. DataApps Corporation 7
Data Patterns
●
Patterns are expressed a regular
expressions
●
Supports common patterns like email,
credit card, SSN etc.
●
Users can add new patterns

8. DataApps Corporation 8
File Formats
●
Text Files
●
All file formats supported via Hive
including Sequence, Avro, Orc, Parquet.
●
Uses Hive Schema to parse input (if
available)

9. DataApps Corporation 9
Scanning
●
Deep Scan vs Quick Scan
●
Incremental scans
●
Scheduled Scans

10. DataApps Corporation 10
Results
●
Collects all sensitive elements
●
Provides a preview of detected elements
by type.
●
Can download the full list with location
pointers.

11. DataApps Corporation 11
Compatibility
●
Hadoop Versions
–
2.3+
●
Hive Versions
–
0.12 +
●
Uses the Hadoop and Hive clients
installed on the machine

12. DataApps Corporation 12
Security
●
Authentication
–
Kerberos, Custom
●
Authorization
–
ACLs to access, administer and view/edit
scans
●
Auditing
–
Records all actions

13. DataApps Corporation 13
Protection of Scans and Results
●
Can enable usage of impersonation to
scan files as the user itself
●
Scan results stored on HDFS are
readable by Chlorine user only
●
Users can view their scans and control
access to the scans and results

14. DataApps Corporation 14
Demo

15. DataApps Corporation 15
On line Demo
●
Visit http://dataApps.io to scan a Hadoop
Cluster using Chlorine
●
Setup Scans against a Hadoop Cluster
preloaded with data, View results
●
Explore Chlorine administrative features

16. DataApps Corporation 16
Getting Chlorine
●
Download from http://dataApps.io
●
Can scan up to 10 Terabytes for free.

17. DataApps Corporation 17
Questions ?