Greenplum - Jacque Istok - Hadoop World 2010

RDBMS and Hadoop
A Powerful Combination
Jacque Istok

You Know Hadoop, But What Is Greenplum?
EMC/Greenplum is an MPP data warehouse system, based off PostgreSQL, with the full capabilities of a traditional RDBMS system. In conjunction with SQL-99 compliance for structured analysis, Greenplum also offers a MapReduce implementation for non structured analysis. In short:
Greenplum ~ Hadoop/Hive

Data in a Typical Enterprise
Data is everywhere – corporate EDW, 100s of data marts, ‘shadow’ databases, spreadsheets, logs, etc
The goal of centralizing all data in a single EDW has proven untenable
EDW
~10% of data
Data Marts and ‘Personal Databases’
~90% of data

Today’s Big Data Challenges
Sources of data and the amount of data to analyze is growing exponentially
Stale data exists because DW solutions cannot ingest the vast amounts of data fast enough
Lack of performance for advanced analytics and complex queries
The number of users and the concurrency of users is increasing rapidly
Security and privacy around the data is both preferred and often mandated

Architecture of HDFS/Hadoop/Hive
Flexible framework for processing large datasets
SQL (subset)
Hive
Process large datasets with support for both SQL and MapReduce
MapReduce
NameNode
NameNode
DataNode
DataNode
DataNode
DataNode
DataNode
Hive Server accepts SQL and dynamically
generates and executes MapReduce code
Materialize data subsets to reduce impact of node failure
…
DataNode servers process analytics close to the data in parallel

Architecture of Greenplum
Flexible framework for processing large datasets
Process large datasets with support for both SQL and MapReduce
SQL
MapReduce
Master
Master
Segment
Segment
Segment
Segment
Segment
Master servers optimize queries
for the most efficient query execution
Interconnect for continuous pipelining of data processing
…
Segment servers process queries close to the data in parallel
MPP Scatter/Gather streaming for fast loading of data

RDBMS Advantages

Common Real World Implementation
Lots ‘O Data

A Cyber-Analytics Data Mart Use Case
Commercial SIEM products struggle with the volumes of data generated in a large enterprise. Non-parallel event processing systems can’t keep up with ingest, user load, etc
Greenplum provides the ability to cost-effectively ingest and store large volumes of sensor data.
Greenplum provides the parallel analytics that support data mining, event correlation, etc, over datasets from TB’s to PB’s in size.
Access and Events
SoR
ETL
GPLoad
Greenplum
Analytics
Data Mart
ODS
SQL
MapReduce
(Perl)
(Python Math Lib)
(R)
BI

Coexistence Approach – Use Case
General Purpose X86 Cluster of Systems
Provides true, complete SQL compliant analytics
Data can be read and written from Hadoop via Greenplum
Store your data structured, unstructured, column or row oriented, compressed, leveraging Index support where appropriate
SQL can be executed, through Greenplum, on data residing within Greenplum as well as data residing within HDFS
MapReduce can be executed through Greenplum in Java, C, Perl, Python or through Java in Hadoop
Designed for rapid analysis of data volumes from less than a terabyte scaling into the petabytes
Compute
Storage
Analytics
Network

Big Data is Complementary to EDW
Greenplum
Enterprise Data Warehouse
Single Source of Truth
1 Logical Model
Heavy data governance and quality
Operational Reporting
Financial Consolidation
MapReduce Analytics Cloud
Source of all raw data (often 10X size of EDW)
Self-service infrastructure to support multiple marts and sandboxes
Rapid analytic iteration, and business owned solutions
Commodity Hardware
Virtual Machines
Public Cloud

http://www.cloudera.com	519
http://www.lifeyun.com	13
unmht://	1

Greenplum - Jacque Istok - Hadoop World 2010

by Cloudera, Inc. on Oct 25, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 533

Statistics

Greenplum - Jacque Istok - Hadoop World 2010 — Presentation Transcript