Securing Data in Hadoop at Uber
•Download as PPTX, PDF•
4 likes•1,102 views
When it comes to data security, Uber’s business has unique needs related to scale, use-case, and technical stacks. This talk will discuss how our data platform team addressed specific challenges in deploying Uber's security requirements for Apache Hadoop, including how we leveraged open source building blocks. We'll share insights on how we augmented our Kerberized Hadoop integration with additional authentications mechanisms as well as our approach to supporting custom authentication in Apache Knox. In particular, we will elaborate Uber’s contributions to Apache Knox, specifically a novel pluggable platform for custom validation of any user request. This talk will also cover how we address table, column, and partition-level access control while ensuring improved developer productivity. In particular, we will explain how we translate RBAC policy into HDFS ACL to control data access, our internal audit platform built to detect and analyze the common security infringements, and real-world examples from our experiences in production. Speakers Mohammad Islam, Staff Software Engineer, Uber Wei Han, Manager, Uber
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Securing Data in Hadoop at Uber
Similar to Securing Data in Hadoop at Uber (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Securing Data in Hadoop at Uber
- 1. Securing Data in Hadoop at Uber TitleMohammad Islam Wei Han
- 2. Speaker Intro • Mohammad Islam – Staff Engineer @Uber – Apache Hadoop contributor, PMC in Oozie & TEZ – Co-Authored O’Reilly book about Apache Oozie • Wei Han – Technical Manager @ Uber – Lead Hadoop Security team
- 3. What is (NOT) covered? • Securing Hadoop data lake at Uber • Focus on technologies – Open source + internal tools • NOT covering all aspects of data security • NOT a legal advice or guidance
- 4. Data Security in Hadoop
- 5. What is Data Security? Prevent unauthorized access to data. Technical focus area in data lake: AAAA Authentication Authorization Auditing Anonymization
- 6. 4 Pillars of Data Security (1/2) 1. Authentication (AuthN) Verify identity of a user 2. Authorization (AuthZ) Access control of data
- 7. 3. Auditing • Post-mortem • Anomaly detection 4. Anonymization • tokenization • Masking etc. 4 Pillars of Data Security (2/2)
- 8. Design Considerations • Secure all access paths to HDFS • Enforcement at the lowest-level • User/group based (AD) access control • Centralized policy store Not at the cost of infrastructure flexibility
- 9. Internal Services BI Tools Workbench ETL Ingestion Machine learning Metadata Hive PrestoSpark WF Scheduler Flink Pinot YARN Mesos At a Glance .. HDFS • Outer layer verifies user identity • Middle layers securely pass the identity • Innermost layer enforces access control Identity verificationPass identityAccess enforcement
- 10. Authentication
- 11. Authentication Overview Hadoop Ecosystem w/ Kerberos Hive, Presto, Spark, Flink etc Non-Hadoop Services Interactive Batch Gateway Machine Apache KNOX Kerberos Custom Token Web Authentication Kinit KerberosKerberos/DT • Kerberos and delegation token (DT) for Hadoop • Custom S2S authN for non-Hadoop Services
- 12. AuthN Protocol Translation - Knox • Why? – Seamless integration among AuthN protocols • Translate custom AuthN protocols to Kerberos • Contributed to Apache Knox – Pluggable AuthN validator for any custom AuthN protocol (KNOX-861, KNOX-869) – Improved monitoring (KNOX-940)
- 13. Impersonation/Delegation • Why? – Hadoop already supports impersonation or doAs • Work on-behalf-of others – Internal authN mechanism doesn’t support it • How? – Utilize Apache Knox – Whitelist the impersonated services using config – Idea borrowed from Hadoop core-site.xml
- 14. Off-label Usage : Delegation Token • Delegation token (DT) is a Hadoop concept • Used DT for authentication only when: Other protocol doesn’t work or is not ready DT for HDFS is already available HTTP REST service • Added support in Presto and few other internal services
- 15. Off-label Usage : Delegation Token : Client Service Service NameNode 1. Get DT w/ Kerb 2. DT as HTTP header 3. Verify DT Summary • (+) Quick and easy to implement • (-) Extra load on NN (caching can address it)
- 16. Authorization
- 18. RBAC Policy to ACLs Design Options 1. Use HDFS plugin (Sentry/Ranger) 2. Set the ACLs directly in HDFS Why Option 2? 1. No overhead (Mem/latency) to NN 2. NN Scale/stability is a concern 3. Challenge : Keep in sync admin Policy Propagator HDFS <Table, permission, groups> setacl <data_dir, permission, groups> Self-serve Platform
- 19. Partition-based Access Control (1/2) • Policy defined at partition level – Usual access control is table-level – Access can change based on time or geography • Example use case: – “events” table is partitioned by date – Policies • By default, employee can only access new events records • Only authorized groups can access events records older than X days
- 20. Partition-based Access Control (2/2) /events/date=2018.06.06 group: employee, authorized_group:r-x /events/date=2017.02.06 group: authorized_group:r-x Table Privilege Group Time restriction events read employee X days events read authorized_group none admin Self-serve Platform Policy Propagator HDFS Set ACL Refresh ACL (periodically) Sample Policies
- 21. Auditing
- 22. Auditing Audit Collection Log Analysis and Monitoring Kafka File access logs Query logs Service access logs HDFS Query Engines (Hive/Presto) Services Producers admin HDFS Query Engines Real-time monitoring
- 23. Anonymization
- 24. Anonymization • Transform any data into unidentifiable form • Loosely used to mention: – Removal – Redaction – Masking – Tokenization Next : Enforce AuthZ through Encryption
- 25. Column-level Access Control • Why? – When only some columns in a table are sensitive and need special access control – Finer grained access control based on level of sensitivity Column C1 C2 C3 …. C15 … C34 … C100 Sensitivity Level 0 8 0 0 6 0 9 0 0 • Challenges – Enforce on common access paths – HDFS doesn’t understand column
- 26. Approach • Enforce in data format (Parquet) level • Encrypt only sensitive columns in HDFS • Access controlled through encryption key management • Different column can have different key for encryption/decryption • Open source activities: PARQUET-1178 And PARQUET-1325
- 27. Key Management Parquet lib Reader (Hive, Spark, etc) C1: plain text C2E: encrypted C1: plain text C2: plain data(has access) error/masked (no access) Writer (Hive, Spark, etc) Parquet lib HDFS C1: plain text (non-sensitive) C2: plain text (sensitive) C1: plain text C2E: encrypted Column-level Access Control
- 28. Conclusion
- 29. Take Away 1. Security scope within Hadoop is expanding • Conventional thinking is being challenged • Need significant changes in the architecture 2. Finer-grain security is must for big data • Column/Partition/Row level access control 3. Security by design is critical • Retrofitting is very hard
- 30. Q&A
Editor's Notes
- This slide is the easiest one. For the sake of time, let me skip it.
- Before going to the details, let’s set the stage first. That means what will discuss and what we will not talk. At Uber, Hadoop is the backbone of our data lake. In this presentation, we will discuss how we achieve security in Hadoop In particular, we will explain how we merge the open source technologies with our in-house tools and servcies There are multiple ways we address data security. However, today we will talk about one of such approach. Also please take the presentation with a grain of salt.
- Source : https://techdifferences.com/difference-between-authentication-and-authorization.html We will discuss more details in later slides.
- Source : https://techdifferences.com/difference-between-authentication-and-authorization.html Meanings are so close /confusing to some people.
- https://theprivacyblog.com/anonymity/data-anonymization-is-hard-this-time-shown-with-nyc-taxi-data/ Deterrent : internal admin or approved users as well. To remove the misuse.
- Authentication is about verifying who you are Authorization is about checking whether you’re allowed to access the data
- This diagram shows the overall architecutre of how authZ works. Spearking of authZ, there're two important parties invovled, admin and user Admin is the one who decides which data can be accessed by who, normally we call this kind of rule, as a policy This diagram show the admin flow. Basically we build a self server platform, which allow admin to manage the policies. Eventually the policy is stored in a policy store, and more importantly policies will be converted to extended ACLs in HDFS. enforcement happends at HDFS, which gaurantees 100% coverage of access control. No matter what type of access(whether it's Hive, Spark, or some new engines adopted), it'll all comes to HDFS.
- Mohammad: bring RBAC concept first. Tell: RBAC is an abstract layer HDFS is physical layer (a set of dirs) converting from abstract->physical Emphasize that we take/implememted #2 Emphasize NN is sensitive service to us. Esp policy will grow #2 has no extra latency/memory overload because it’s offline Now let's look at how we covert the policy to ACLs in HDFS. Again let's bring up this diagram. As admin, you can define a policy, which is essentially is a tuple table, permission type(whether it's read or write), and which group can have the access Then a policy propagator will propagate the policy, to all HDFS files, which essentially setACL on all the table's files. Now in order to utilize ACls for enformcene, we had two options. First is we can utilize some HDFS plugins The 2nd option is what we presetned here, basically we set the ACL directly Eventually we chose 2nd option. The primary reason is #1 will have some overhead to NN. Becaue option 1 requires NN to cache all table's metadata and policy in memory, which can be a big deal when your table growsl. In the other hand, NN's scalability is a big and ongoing challengeing so we don't want to add extra overhead to NN. However in practice option2 also have challenges. The most challeging part is keep the policy in sync between policy store and HDFS. We have to build lots of monitoring, and even reconcilation to make these in sync.
- Talk about examples (time, geo) So far we only talked about table level access control. But in some use cases, Now let's look at partition level access control. Usually access control happens at table level. However in some scenarios, access can change based on time or geography. Let's look at an concrete example
- Let’s look at a realistic example how we achive this (time based refresh can be shorter) same as before, a policy has table, privilege, and group. However in this scneario, admin can add a time restriction. This is the same policy talked about. Employees's access has a time restriction of X days, meaning employee can only access events happens in the past X days. The 2nd policy indicates legal team doesn't have any time restriction. They can access all data. Now same as the flow, the policy propagator now will set different ACLs for different partitions, depending on how old the partition is. So in this example, partitiion 2018.06.06, the ACL indicates both employee and legal has R access. However the older parition 2017.02.06, only the legal team has access. Also note we have to keep reresh the ACL according to the polilcy, when parttion becomes older.
- (reduce time) Flink job
- Source: https://simplicable.com/new/data-anonymization Don’t need to mention all techs Today in this presentation, we’ll only focus on talk use encryption to achieve anoymizaiton in next slide
- Collevrating with IBM and other companies thru Apache community
- <shorter READ part> concrete example. in This example, c1 is non-sensitive, c2 is senstive When the writer writes the data to HDFS(whether it's Hive, Spark, whateer), for c1, it'll writes plain text to HDFS. For c2, we'll write encrypted data to HDFS. Now look at reader part. Whenever any reader reads the data, c1 will always be plain text. But for c2, on the parquet reader side, we'll check whether the user has access to the sensisitive column c2 or not. If it does, we'll retrive the encryption key and decrypt the data. Otherwise, if the user doesn't have access, we'll return an error or mask the data