At a Big Data Warehousing Meetup, George Corugedo, CTO of RedPoint Global demonstrated how to use your big data platform for data integration, data quality and identity resolution to provide a true 360 degree view of your customer on Hadoop using the RedPoint product. For more information or questions, please contact us at www.casertaconcepts.com.

1. Data Quality in the Data Hub
February
2015

2. 2 © RedPoint Global Inc. 2015 Confidential
Overview of RedPoint Global
Launched
2006
Founded
and
staffed
by
industry
veterans
Headquarters:
Wellesley,
Massachuse@s
Offices
in
US,
UK,
Australia,
Philippines
Global
customer
base
Serves
most
major
industries
MAGIC
QUADRANT
Data
Quality
MAGIC
QUADRANT
MulNchannel
Campaign
Management
MAGIC
QUADRANT
Integrated
MarkeNng
Management

3. 3 © RedPoint Global Inc. 2015 Confidential
Cloudera Stack

4. 4 © RedPoint Global Inc. 2015 Confidential
Andrew Brust, GigaOm Research

5. 5 © RedPoint Global Inc. 2015 Confidential
There is lots of Hype Out There

6. 6 © RedPoint Global Inc. 2015 Confidential
Don’t believe the Marketing Hype

7. 7 © RedPoint Global Inc. 2015 Confidential
Purpose of Data Quality
Create correct and consistent
data across the enterprise that
fosters trust in information and
acceleration of growth.

8. 8 © RedPoint Global Inc. 2015 Confidential
New Data Straining Current Architectures
Unstructured
documents,
emails
TransacNonal
data
Server
logs
SenNment,
web
data
GeolocaNon
Sensor,
machine
data
Clickstream
Hierarchical
data
OLTP,
ERP,
CRMMaster
data
2.8
ZB
in
2013
85%
from
new
data
types
15x
Machine
Data
by
2020
40
ZB
by
2020
Source: IDC

9. 9 © RedPoint Global Inc. 2015 Confidential
Data Hub for MDM
Data
Hub
1Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
Ÿ
n
YARN
Production RDBMS
Databases
Data
IngesNon
Specialized Analytic
Databases & Caches
Any analytics
Any reporting
Predictive Analytics
Clustering
Profiling
AnalyNcs
Marketing Automation
Real Time Personalization
Omni-Channel Optimization
Digital and Traditional Channels
InteracNon
Systems
Data
Quality
Processing
Persistent Entity Resolution, Linkage and Keying

10. 10 © RedPoint Global Inc. 2015 Confidential
How About MDM on a Data Lake?
• Severe
shortage
of
Map
Reduce
skilled
resources
• Inconsistent
skills
lead
to
inconsistent
results
of
code
based
soluNons
• Nascent
technologies
require
mulNple
point
soluNons
• Technologies
are
not
enterprise
grade
• Some
funcNonality
may
not
be
possible
within
these
frameworks
Challenges
to
Data
Lake
Approach
• Data
is
ingested
in
its
raw
state
regardless
of
format,
structure
or
lack
of
structure
• Raw
data
can
be
used
and
reused
for
differing
purposes
across
the
enterprise
• Beyond
inexpensive
storage,
Hadoop
is
an
extremely
power
and
scalable
and
segmentable
computaNonal
plaorm
• Master
Data
can
be
fed
across
the
enterprise
and
deep
analyNcs
on
clean
data
is
immediately
enabled
Benefits
of
a
Hadoop
Data
Lake

11. 11 © RedPoint Global Inc. 2015 Confidential
Key Functions for Master Data Management
Master
Key
Management
ETL
&
ELT
Data
Quality
Web
Services
IntegraNon
IntegraNon
&
Matching
Process
AutomaNon
&
OperaNons
• Profiling,
reads/writes,
transformaNons
• Single
project
for
all
jobs
• Cleanse
data
• Parsing,
correcNon
• Geo-­‐spaNal
analysis
• Grouping
• Fuzzy
match
• Create
keys
• Track
changes
• Maintain
matches
over
Nme
• Consume
and
publish
• HTTP/HTTPS
protocols
• XML/JSON/SOAP
formats
• Job
scheduling,
monitoring,
noNficaNons
• Central
point
of
control
• Meta
Data
Management

12. 12 © RedPoint Global Inc. 2015 Confidential
Overview - What is Hadoop/Hadoop 2.0
Hadoop
1.0
• All
operaNons
based
on
Map
Reduce
• Intrinsic
inconsistency
of
code
based
soluNons
• Highly
skilled
and
expensive
resources
needed
• 3rd
party
applicaNons
constrained
by
the
need
to
generate
code
Hadoop
2.0
• IntroducNon
of
the
YARN:
“a
general-­‐purpose,
distributed,
applicaNon
management
framework
that
supersedes
the
classic
Apache
Hadoop
MapReduce
framework
for
processing
data
in
Hadoop
clusters.”
• Mature
applicaNons
can
now
operate
directly
on
Hadoop
• Reduce
skill
requirements
and
increased
consistency

13. 13 © RedPoint Global Inc. 2015 Confidential
RedPoint Data Management on Hadoop
ParNNoning
AM
/
Tasks
ExecuNon
AM
/
Tasks
Data
I/O
Key
/
Split
Analysis
Parallel
SecNon
YARN
MapReduce

14. 14 © RedPoint Global Inc. 2015 Confidential
Resource
Manager
Launches
Tasks
Node
Manager
DM
App
Master
DM
Task
Node
Manager
DM
Task
DM
Task
Node
Manager
DM
Task
DM
Task
Launches
DM
App
Master
Data
Management
Designer
DM
ExecuOon
Server
Parallel
SecNon
Running
DM
Task
1
2
3
RedPoint DM for Hadoop: Processing Flow

15. 15 © RedPoint Global Inc. 2015 Confidential
Reference Hadoop Architecture
Monitoring and Management Tools
Management
MAPREDUCE
REST
DATA REFINEMENT
HIVEPIG
HTTP
STREAM
STRUCTURE
HCATALOG
(metadata services)
Query/Visualization/
Reporting/Analytical
Tools and Apps
SOURCE
DATA
- Sensor Logs
- Clickstream
- Flat Files
- Unstructured
- Sentiment
- Customer
- Inventory
DBs
JMS
Queue’s
Fil
es
Fil
esFiles
Data Sources
RDBMS
EDW
INTERACTIVE
HIVE Server2
LOAD
SQOOP
WebHDFS
Flume
NFS
LOAD
SQOOP/Hive
Web HDFS
YARN
Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ
Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ
Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ
Ÿ Ÿ
Ÿ Ÿ
Ÿ n
1 Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ
Ÿ
Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ ŸŸ
Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ ŸŸ
Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ Ÿ
HDFS
RedPoint Functional Footprint

16. 16 © RedPoint Global Inc. 2015 Confidential
>150
Lines
of
MR
Code
~50
Lines
of
Script
Code
0
Lines
of
Code
6
hours
of
development
3
hours
of
development
15
min.
of
development
6
minutes
runNme
15
minutes
runNme
3
minutes
runNme
Extensive
opNmizaNon
needed
User
Defined
FuncNons
required
prior
to
running
script
No
tuning
or
opNmizaNon
required
RedPoint
Benchmarks – Project Gutenberg
Map
Reduce
Pig
Sample
MapReduce
(small
subset
of
the
entire
code
which
totals
nearly
150
lines):
public
static
class
MapClass
extends
Mapper<WordOffset, Text, Text, IntWritable> {
private
final
static
String delimiters =
"',./<>?;:"[]{}-=_+()&*%^#$!@`~ |«»¡¢£¤¥¦©¬®¯±¶·¿";
private
final
static
IntWritable one = new
IntWritable(1);
private
Text word = new
Text();
public
void
map(WordOffset key, Text value, Context context)
throws
IOException, InterruptedException {
String line = value.toString();
StringTokenizer itr = new
StringTokenizer(line, delimiters);
while
(itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
Sample
Pig
script
without
the
UDF:
SET
pig.maxCombinedSplitSize 67108864
SET
pig.splitCombination true
A = LOAD
'/testdata/pg/*/*/*';
B = FOREACH A GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS
word;
C = FOREACH B GENERATE UPPER(word) AS
word;
D = GROUP
C BY
word;
E = FOREACH D GENERATE COUNT(C) AS
occurrences, group;
F = ORDER
E BY
occurrences DESC;
STORE F INTO
'/user/cleonardi/pg/pig-count';

17. 17 © RedPoint Global Inc. 2015 Confidential
Data Lake Architecture for MDM

18. 18 © RedPoint Global Inc. 2015 Confidential
Recommendations for Data Quality
• There
is
a
gap
between
current
use
and
the
mainstream
• Don’t
believe
the
hype;
there’s
plenty
of
it
• Data
Quality
creates
trust
in
informaNon
which
enables
confident
and
nimble
decision
making.
• Look
for
broad
enterprise
apps
that
have
solved
the
parallel
scalability
problem
• Consider
a
Data
Hub
approach
for
Data
Quality
for
maximum
flexibility
and
scalable
performance