Filling the Data Lake

Filling the
Data Lake
June 29, 2016
Chuck Yarbrough
Sr Director, Solutions Marketing and Management
@cyarbrough
Mark Burnette
Enterprise Sales Engineer @MarkCBurnette

© 2015, Pentaho. All rights reserved. pentaho.com. Worldwide +1 (866) 660-75552
Emerging Big Data Use Cases
Improve operational
effectiveness
Machines/sensors:
predict failures, network
attacks
Financial risk management:
reduce fraud, increase
security
Reduce data warehouse cost
Improve customer
experience
Build a 360° view to fully
understand and serve the
customer
Drive personalized and
adjusted interaction
Use automated
recommendations logic
Drive incremental
revenue
Predict customer
behavior across all channels
Understand and
monetize customer behavior
Begin to monetize data
as a service

Spectrum of Big Data Use Cases
Entry
Transform
Advanced
Optimize
Data
Warehouse
Optimization
Streamlined
Data
Refinery
Big Data
Exploration
Customer
360 Degree
View
Harnessing
Machine &
Sensor Data
Next
Generation
Applications
Internal Big
Data as a
Service
On-Demand
Big Data
Blending
Big Data
Predictive
Analytics
Use Case Complexity
BusinessImpact
Monetize My
Data
Data
Warehouse
Optimization
Data
Warehouse
Optimization
Streamlined
Data
Refinery
360 Degree
View
Big Data
Onboarding
Filling the
Data Lake

Administration Security
Lifecycle
Management
Data
Provenance
Dynamic Data
Pipeline Monitoring Automation
Data Pipeline
Data Engineering
Managing and Automating the Pipeline
Data Engineering AnalyticsData Preparation
Data
Lake

The Data Swamp

The Data Lake

Does Hadoop Have to be Hard?
Empower team
members to
integrate and
process Hadoop
Data
Establish a
modern data on
boarding process
that is flexible and
scalable
Deliver governed
analytic insights
for large
production use
bases
Things that can help ease the pain

Proper Care and Feeding of the Data Lake

More Data, More Problems
Even with good integration tools, major data onboarding
projects can be painful:
User Challenges
§  Repetitive manual design
§  Very time-consuming
§  Difficult to maintain
Business Challenges
§  Takes too long
§  Business deadlines at risk
§  Opportunity cost

How do we effectively scale data pipelines to accommodate
exploding data sources, volumes, and complexity?
Have you ever had the pleasure of…
Migrating hundreds of sources between systems?
Enabling business users to onboard a variety of data themselves?
Ingesting hundreds of changing data sources into Hadoop?

Modern data onboarding is more than
just “dumping data” – it includes:
Managing a changing array of data sources
Establishing repeatable processes at scale
Maintaining control and governance

CSV
RDBMS
Data On Boarding
Filling the Data Lake
Ingest Procedures
Disparate Data Sources Integration Processes Transformations
Hadoop
AVRO

CSVCSV
RDBMS
Data On Boarding at Scale
RDBMS
Disparate Data Sources Integration Processes Transformations
RDBMS
Ingest Procedures
Hadoop
AVRO

Filling the Data Lake
A Modern Data Onboarding Blueprint
Streamline data
ingest from wide
variety of source data
Reduce dependence
on hard coded data
movement procedures
Simplify regular data
movement at scale
into Data Lake

CSVCSV
RDBMS
Dynamic ELT
Ingest Templates
Hadoop
RDBMS
Disparate Data Sources Dynamic Integration Processes Dynamic Transformations
RDBMS
Pass metadata in at run time
to generate jobs on the fly
(metadata injection)

CSV
CSV
RDBMS
Templated workflows
RDBMS -> AVRO
Template
Hadoop
RDBMS
RDBMS
CSV -> AVRO
Template
CSV -> HDFS
Template

Variety – different metadata, one template
Hadoop
CSV -> AVRO
Template

Key Takeaway
Managing
ELT and ELT
procedures
Managing
Metadata
Metadata Injection

RDBMS Ingestion
Automated
Metadata
Extraction
Extract table and store in AVRO
§  Database connection details
§  Table(s)
§  Field names (if available)
§  Data types
§  String length
§  Mask for numbers and dates
§  …

Option 1: Ingest RAW files into
HDFS (no parsing)
§  Path to CSVs
CSV Ingestion
Option 2: Parse and store in AVRO
§  Path to CSVs
§  Delimiter
§  Field names (if available)
§  Data types
§  String length
§  Mask for numbers and dates
§  …
Automated
Metadata
Extraction

Key Takeaway
ELT development
DAYS
Provisioning
MINUTES
Automated Metadata Extraction

Key Takeaways
Template-based
Data Integration
Manage metadata
vs.
ELT procedures
Automated
Metadata
Extraction
Provide minimum
required
configuration
Reduce Risk
Maintain an
organized,
standardized, &
clean, data lake
Data Onboarding Blueprint

Learn more about Big
Data Onboarding at
Pentaho.com
Download Pentaho
Platform at
Pentaho.com
What Next?

Filling the Data Lake

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Filling the Data Lake

Similar to Filling the Data Lake (20)

More from DataWorks Summit/Hadoop Summit

More from DataWorks Summit/Hadoop Summit (20)

Recently uploaded

Recently uploaded (20)

Filling the Data Lake