This document discusses operationalizing a data lake for advanced analytics. It describes how a data lake platform can provide end-to-end capabilities for managing the data supply chain from source to consumer. This includes batch and streaming ingestion, automatic discovery, data quality, security, lifecycle management, metadata cataloging, transformations, and self-service data preparation. A reference architecture is presented showing how raw and refined data zones can feed a trusted zone to provide role-based access to business analysts, researchers, and data scientists. Machine learning techniques are also described for integrating data silos, matching records, and classifying sensitive data.

1. Operationalizing your Data Lake:
Get Ready for Advanced Analytics
October 22nd , 2017
Parth Patel | Big Data Solutions Engineer
ppatel1@zaloni.com

2. 2
Industry-leading enterprise
data lake management,
governance and
self-service platform
Expert data lake
professional services
(Design, Implementation,
Workshops, Training)
Solutions to simplify
implementation
and reduce business risk
Enabling the data-powered enterprise
Zaloni Confidential and Proprietary

3. 3 Zaloni proprietary – do not duplicate without permission
Increased
Agility
New
Insights
Improved
Scalability
Data lakes are central to the modern data architecture

4. 4 Zaloni Proprietary
Data architecture modernizationTraditionalModern
Data Lake
Sources ETL EDW
Derived
(Transformed)
Discovery Sandbox
EDW
Streaming
Unstructured Data
Various
Sources
Data Discovery
Analytics BI
Data Science
Data Discovery
Analytics BI

5. Zaloni Confidential and Proprietary - Provided under NDA
5 Zaloni Proprietary
0% of market
Optimize
Self-Organizing Data Lake
• Self-improving data
lake via machine
learning algorithms
• True democratization
of big data and
analytics
• Intelligent data
remediation and
curation
• Recommended Data
Security, and
Governance policies
• Lights out business
operations optimized
for business success
2% of market
Automate
Responsive Data Lake
• Self-Service Ingestion
& Provisioning
• 360 View of Customer,
Product, etc
• Enterprise Data
Discovery
• Operationalize
analytical models into
business fabric
• Enables immediate data
impact on business
operations
Manage
10% of market
Managed Data Lake
• Acquire useful data from
across the enterprise
• Improved visibility and
understanding via
managed Ingestion of
data and metadata
• Ensure security and
privacy of sensitive data
• Operationalize
data at scale
• Leverage enterprise
governance &
security policies
• Scalable production data
lake for new and improved
business insights
22% of market
Store
Data Swamp
• Hadoop on premises
or in the Cloud
• Limited visibility and
usability of data
• Limited corporate
oversight & governance
• Sandbox or Dev
Environments
• Ad hoc and incremental
growth of big data
applications
• Ad-hoc and exploratory
insights for individual
use cases
Zaloni Big Data Maturity Model
Stage:
Characteristics:
Descriptor:
Stage Today:
Business
Impact:
Ignore
66% of market
• Emphasis on
structured data
• Limited ability to
leverage data at
scale
• Business emphasis
on retrospective
reporting and
analysis
• Strong governance
and security policies
• Slow to
accommodate
business changes
Data Warehouse
Value Realized

6. 6 Zaloni proprietary – do not duplicate without permission
Managing the Data Supply Chain from Source to Consumer
CONSUMERS
Business Analysts
Researchers
Data Scientists
Applications
• Data Lake Management Platform
• A software solution for data lake management that enables enterprise-wide scalability
• Provides end to end capabilities
Self-Service
Data
Data Lake Management Platform
Enable Govern Engage
Batch ingestion
Streaming
Ingestion
Auto
discovery
Data Quality
Data Privacy and
Security
Data Lifecycle
Management
CatalogMetadata
Management
Operationalize
Transformations
Self-Service
Data Preparation
PRODUCERS
File Data
Streaming
Relational
On-premise

7. 7 Zaloni Proprietary
Data Lake Reference Architecture
• Data required for LOB specific views - transformed
from existing certified data
• Consumers are anyone with appropriate role-based access
• Standardized on corporate governance/ quality policies
• Consumers are anyone with appropriate role-based access
• Single version of truth
Transient
Landing Zone
Raw
Zone
Refined Zone
Trusted Zone
Sandbox
Data Lake
• Temporary store of
source data
• Consumers are IT,
Data Stewards
• Implemented in highly
regulation industries
• Original source data
ready for consumption
• Consumers are ETL
developers, data
stewards, some data
scientists
• Single source of truth
with history
• Data required for LOB specific views - transformed
from existing certified data
• Consumers are anyone with appropriate role-based access
Sensors
(or other time series data)
Relational Data
Stores
(OLTP/ODS/DW)
Logs
(or other unstructured
data)
Social and
shared data

8. 8 Zaloni Proprietary
Machine learning for data lake implementations
Loyalty
Customer
Service
TransactionsMarketing
3rd Party
● Easily integrate data silos
● Probabilistic data matching and record
linkage
● Automatically classify, encrypt/mask
PII/Sensitive data for regulatory
compliance
Integrate Data
Silos

9. 9 Zaloni Proprietary
• Extend data lake beyond Hadoop
• Catalog traditional sources
• Ingest datasets without IT
• Prepare & provision data to your tool of
choice
Increasing data lake adoption through self-service
Self-Service Data
Preparation

10. 10 Zaloni Proprietary
DON’T GO IN THE
DATA LAKE
WITHOUT US
Questions?