Multi-Tenancy in Data Lakes are on the rise. When looking at multi-tenancy from the lens of data governance, a lot is changing the landscape, and the way we have been operating with respect to the governance model probably needs a rethink. It is time to think of Governance and its various entities as a first-class citizen in data architecture and bake it as part of the platform. We will look at the various aspects of governance, extending to accommodate the growing compliance and regulatory requirements and suggestive architectural approaches to realize the same.

1. Data Governace & MultiTenancy
A Tech Perspective
Sathish K S
VP Engineering
Zeotap

2. Intro

3. ZEOTAP
 Customer Intelligence Platform – SAAS, DAAS & Combined offering
 Enables Brands to better understand their customers
o 360 View
o Identity resolution
o Activation
 Data Assets – People centric including from Telcos
 Full Privacy/GDPR compliant
 Catering to Ad-Tech and MarTech

4. DATA GOVERNANCE
 Data Principles – Data as an asset and linkage with business
 Data Quality – Need of standards of the asset
 Meta Data - Semantics of the asset
 Data Access - Permissions
 Data Lifecycle – States and management
KHATRI & BROWN - 2010

5. DATA GOVERNANCE
 Cross-Functional Framework
 Manage Data as an Asset
o Access Rights
o Processing Rights
o Activation
 Policies & Standards
 Compliance Adherence
 Monitoring and Auditing
 Acts on Information and not on Data
 You Cannot Govern What You Don’t Know

6. Features

7. DATA GOVERNANCE - General
 Explore
 Provision to assign ownership
 Provision for knowledge attribution - lineage
 Provision for access rights
 Categorization and Use case
 Apply Rules
 Metering outflow
 Derivation / Inference history
Goals
 Democratize
 Secure
 Compliance
 Quality

8. DATA GOVERNANCE - Accommodate
 Changing Regulations
 More variety in data
o AI Generated
o Biometrics PII
 Country
 Data ownership flux

9. Basic Tech Building Blocks
Data Assets
Inventory
• Location
• Asset Type
• Category
• Version
• Source
Data Inventory
• Schema
• Data Types
• Tags
• Cardinality
• Values
• Quality Stats
Data Security
• Access Control
• Encryption
• Masking
• Erasure
Data Lineage
• Knowledge
source
• Timestamp
• Weights

10. Basic Tech Building Blocks
Data Quality
• Stats
• Freshness
• Density
• Anomaly
• Dedup
• Quality Stats
Policy
Management
• Compliance
• Consent
• Thresholds
• Lifecycle
• Actions
Audit
&
Monitoring
• Notifications
• Logging
• Analytics

11. Throw in Multi-Tenancy

12. DATA GOVERNANCE – MULTI-Tenancy
 Organizational
 Vertical
o Regulations
 Data Domain
o Data Profile / Quality
o Security
o Lifecycle
o Processing
o Storage
o Metadata

13. DATA GOVERNANCE - MULTITenancy
 Application or usage Context
o Analytics
o Derivations
o Marketing
o Personalization
 Multi-Tenant Architectural Implications
o Shared Everything
o Shared Nothing
o Shared Stateless Control Planes

14. In Essence
DATA GOVERNANCE
IS BECOMING
A BIG DATA PROBLEM IN ITS OWN

15. Building The Bits

16. Catalog Evolution
What is in my
data?
- Schema
Who gave the
data?
- Lineage
How do you
describe?
- Raw or
Inferred
Where all is it
stored, when?
- Location
- Version
How is it
used?
- Purpose
Who owns it?
- Access
Who uses it?
- Roles
What rules
govern?
- Governance

17. POLICY Management & Catalog
 Policy and Actions
 Policy Types – schema, value, record…
 Action – Drop action, Alert action, Null Action…
 Hierarchy support
 Runtime parameters like blacklist & whitelist
 Thresholds
 F(Policy) + G(Param) = Action to be actuated

18. Lifecyle & Ownership
Collect
• Source
• Provider
Process
• Processor
• Auditor
Analyze
• Analyst
• Validator
Share
• Consumer
• Sharer
EOL
• Source
• Provider
Owner responsible for assigning permissions for usage

19. Continuous Quality Measurement
Problem
 Quality linked to Usage or Value Creation
 Monitor, Detect, Propagate
Tools
 AWS Deequ – cloud giving first class citizenship to quality. Open sourced
 Apache Griffin - Platform style more than library style
 Datacleaner – https://datacleaner.org/
 Great Expectations - https://greatexpectations.io/ - You may be surprised!

20. Lineage Problem – DAAS Multitenancy
 Who is the knowledge contributor?
 Challenge
o Multiple Data sources
o Conflicting attributes
o Derivation models

21. Catalog – On Steroids
 Interface for search
 Lookups
 Multi-relational
 Transitivity
 Updates
o Registration
o Onboarding
o Processing output
o Futuristic AI / ML based

22. To Explore
 Tooling is maturing at rapid pace
Softwares beyond cloud natives
 Poster child – Amundsen from Lyft
 Challenger – Apache Atlas
o Extensible Entity and Type System
o Backed by GraphDB (Janus) & Elastic
 Upcoming
o Atlan – Definitely Check This out - https://atlan.com/
o Marquez Project – Also this - https://marquezproject.github.io/marquez/

23. Tools Evaluation
 Storage integrations
 Intelligent crawling
 APIs to push metadata from processing systems
 Event based integrations
 Self serve UI & APIs
 Bulk export capabilities for AI/ML on meta

24. Security - Infra
 Principles of Zero Trust and Positive Security
 Perimeter
o VPN in cloud
o Apache Knox
 Access – Attribute based access control
 Masking
 Encryption/Hashing
 Key Management – important in data exchange

25. Audit Service
■ Define Grammar – Queryable using popular SQL on Hadoop
■ Cloud storage – S3, GCS
■ Logs grammar to include
○ ViolationType
○ Product Code
○ Dataflow stage
○ Action taken
○ Action Timestamp
○ Violation Timestamp
○ ++

26. Declarative Data Workflows
Problem : Governance – react to changes, dynamic, extensibility
 Invest in declarative data pipelines and workflows (workflow as a code)
 Invest in Functional repository of Pure Functions
Tools
 Poster child – Airflow
 Upcoming
o Prefect – parameterized dynamic workflows, irregular / no schedule - https://github.com/prefecthq/prefect
o A-La-Mode – emphasis on output versioning and success detection - https://github.com/binaryaffairs/a-la-mode

27. Accumulate Function Repo
Pure Function Repositories with Functional
Assets to perform
 Enrich the data
 Standardize
 Normalize
 Apply a Rule
 Find Pattern (Regular Expression)
 Sort
 Filter
 Merge
 Transpose
 Parse
 Transform Data Types
 Missing Data Handling

28. Ref Architectures

29. Where is Zeotap
 SAAS – shared nothing
 DAAS – shared everything
 ID resolution product – shared control plane
 Provisioning System
 Heavily borrowed from SDN concepts
 Testable IAAC with
o SLAs for slowly created assets & quick assets
o Error handling
o Transactional
o Ease of Billing

30. Path To Self Serve
 Provisioning System
o Support – Shared Nothing, Shared Control Plane, Shared Resources style
o Cloud APIs
 Workflow as a Service
o In house control plane – Mr Sheperd & Kingpin based on Apache Camel
o Abstraction over Airflow, Oozie and Livy
 Self serve Compliance and Consent
o Supported Regulations – GDPR, CCPA
o Frameworks – TCF, TCF 2.0
o Add your own rules (DSL)

31. Provisioning System - Evolving

33. Thank You
● Share this session on social media using #DataLakeSummit
● Interact with other attendees on Slack at datalake-summit.slack.com