Critical data management practices for easy and unified data access that is secured and meets regulatory compliance by Amandeep Khurana, CEO, Okera The desire to stay competitive, deliver better customer experiences and gain greater business agility is driving companies to move their data platforms to the cloud. Data analysts and scientists want modern, advanced analytics that are self-service. Companies who are well down the path to the cloud now have multiple projects involving various parts of the company but often without any notion of access, governance or security common across these projects. This quickly opens the door to risk and confusion. This session will focus on critical data management practices that companies should adopt to achieve the desired benefits and avoid the pitfalls that could lead to complex expensive architectures and unattainable enterprise or regulatory governance requirements.

1. © Okera, Inc. – Confidential and Proprietary | 1
©2018 Okera, Inc. – Confidential and Proprietary 1
August 11th 2018
Data Management Best Practices
Amandeep Khurana

2. ©2018 Okera, Inc. – Confidential and Proprietary 2
Who I am
• Dad to a 7 week old girl
• Dog-dad to a 2 yr old rescue
• Foodie
• Music lover
• Love solving hard problems
• Inefficiency killer
About Me
What I do
• CEO, Co-Founder, Okera
• Principal Solutions Architect @
Cloudera
• Software Engineering @ AWS EMR
• Co-author, HBase in Action
• Help people get value from
technology

3. © Okera, Inc. – Confidential and Proprietary 3
Focus for today:
Data Lakes in the Cloud

4. ©2018 Okera, Inc. – Confidential and Proprietary 4
Agenda 1. Core tenets of a modern data lake
2. Why is cloud different?
3. Data Management topics

5. ©2018 Okera, Inc. – Confidential and Proprietary 5
This is not a vendor pitch or endorsement

6. © Okera, Inc. – Confidential and Proprietary 6
Data Lake is a capability
It’s not a technology stack

7. ©2018 Okera, Inc. – Confidential and Proprietary 7
Core Tenets 1. Separation of storage and compute
2. Elastic
3. Multi-tenant
4. Flexibility in compute tools
5. Flexibility in storage systems
6. Flexibility in infrastructure
7. Dynamically on-board new users and workloads
8. Secure, Governed, Monitored
• Not just the technology
• Scale independently
• Add/remove independently
• Making changes at one layer
shouldn't affect the other
• Regardless of which tool you
use, same catalog, same
security, same auditability, same
monitoring.

8. ©2018 Okera, Inc. – Confidential and Proprietary 8
• Fundamental paradigm shift: Everything is a service
• The stack is getting deconstructed. Including the data warehouse
• Elasticity
• Pay based on usage
• Decision making: IT → Lines of Business (end users)
Cloud v/s Data Center: what’s different?

9. ©2018 Okera, Inc. – Confidential and Proprietary 9
Data
Governance
topics
1. Metadata management
2. Authentication
3. Access Control and Obfuscation
4. Audit
5. Misc topics

10. ©2018 Okera, Inc. – Confidential and Proprietary 10
Primer (Critical to get right)
3 kinds
• Technical (schemas, file locations, size, owner)
• Operational (transformations, queries, performance,
timing, how fresh)
• Business (semantic meaning, access policies, tags)
Metadata

11. ©2018 Okera, Inc. – Confidential and Proprietary 11
Approach
Wrong: Get a crawler to find sensitive info (fingerprint)
Wrong: Get a catalog to classify and govern data
Wrong:
Right: Build technical -> operational -> business. Think
holistically. Afterthoughts are expensive.
Wrong: ”It’s schema on read”
Right: It certainly is schema on read. Reader needs a
schema. Schemas enable sharing.
Metadata

12. ©2018 Okera, Inc. – Confidential and Proprietary 12
Where?
Wrong: Multiple metastores. Let users read RDS.
Right: Put API in front of database
Wrong: Use wiki to store dataset locations and schemas
Right: Build a metastore and integrate in your workflow
Metadata

13. ©2018 Okera, Inc. – Confidential and Proprietary 13
Source of identity
Wrong: IAM
Right: AD / SSO for every systems. Single source
Granularity
Wrong: IAM roles
Right: Users and Groups
Protocols
Wrong: Kerberos
Right: Tokens / OAuth etc
Authentication

14. ©2018 Okera, Inc. – Confidential and Proprietary 14
Mechanism
Wrong: IAM on bucket policies
Right: Role based
Where?
Wrong: S3 level (or any storage level)
Wrong: Compute engine
Right: Independent of storage and compute. Remember: Flexibility
and separation of storage and compute. It’s “data” access controls.
Build independent policy engine that works with multiple tools.
Access Control and Obfuscation

15. ©2018 Okera, Inc. – Confidential and Proprietary 15
Who?
Wrong: Central team
Right: Distributed stewardship
How?
Wrong: Create copies with slices of data for different users (think FGAC, GDPR
consent)
Right: Single copy, control access on top.
Wrong: Obfuscate data and then store it
Worse: Throw away key
Worse: Hard-code key into application code
Right: Store full fidelity. Control access. Obfuscate at read time. Use case specific.
Access Control and Obfuscation

16. ©2018 Okera, Inc. – Confidential and Proprietary 16
When to build?
Wrong: After everything else
Right: Architect from the get go. Auth, Auth and Audit go
together
Wrong: Oh, I have this info. Cloudtrail + EMR logs
Right: Audit is an always on capability. Build reports. Ask
questions of the audit trail. Don’t leave for later.
Wrong: Build a chargeback system
Right: Build a rich audit trail system that gives visibiility
Audit

17. ©2018 Okera, Inc. – Confidential and Proprietary 17
Who’s responsible?
Wrong: Central team
Right: Distributed stewardship
Wrong: Deletes and updates by reading entire partition/dataset, recreating
it, writing it in a new location, alter partition to change location (CDC,
GDPR)
Wrong: Use a different storage systems (HBase, Kudu etc)
Right: Build views to reconcile on the fly. Compact periodically
Wrong: Crawl the lake and find stuff
Right: Catalog as a part of the ingest / creation process. Don’t just fill the
lake
Misc Topics

18. ©2018 Okera, Inc. – Confidential and Proprietary 18
Tool Selection
Wrong: Same from data center should work
Right: Build cloud based services as capabilities
Wrong: Oh we should just build this
Right: Focus on your core competency. Buy stuff. It’s
cheaper
Misc Topics

19. ©2018 Okera, Inc. – Confidential and Proprietary 19
• Cloud and separation of storage and compute is a powerful
paradigm
• Think about data management holistically
• Approach as capabilities and services in the cloud
• Build the foundations first
• Use the right abstractions and frameworks
Summary

20. © Okera, Inc. – Confidential and Proprietary 20
Scaling governance enables self-service and agility.

21. Thank you
ak@okera.com