Pivotal HD and Spring for Apache Hadoop

2© Copyright 2013 Pivotal. All rights reserved. 2© Copyright 2013 Pivotal. All rights reserved.
Hadoop and Pivotal HD
April 23, 2013

3© Copyright 2013 Pivotal. All rights reserved.
About the speakers
Adam Shook
– Technical Architect for Pivotal
– 2+ years Hadoop experience
– Instructor for Hadoop-based courses
Mark Pollack
– Spring committer since 2003
– Founder of Spring.NET
– Lead Spring Data family of projects

Agenda
What is Hadoop?
Pivotal HD
HAWQ
Spring for Apache Hadoop
Questions

5© Copyright 2013 Pivotal. All rights reserved. 5© Copyright 2013 Pivotal. All rights reserved. 5© Copyright 2013 Pivotal. All rights reserved.
What is Hadoop?

Why Hadoop is Important?
Delivers performance and scalability at low cost
Handles large amounts of data
Stores data in native format
Resilient in case of infrastructure failures
Transparent application scalability

Hadoop Overview
Open-source Apache project out of Yahoo! in 2006
Distributed fault-tolerant data storage and batch processing
Linear scalability on commodity hardware

Hadoop Overview
Great at
– Reliable storage for huge data sets
– Batch queries and analytics
– Changing schemas
Not so great at
– Changes to files (can‟t do it…)
– Low-latency responses
– Analyst usability

HDFS Overview
Hierarchical UNIX-like file system for data storage
– sort of
Splitting of large files into blocks
Distribution and replication of blocks to nodes
Two key services
– Master NameNode
– Many DataNodes
Secondary/Checkpoint Node

How HDFS Works - Writes
DataNode A DataNode B DataNode C DataNode D
NameNode
1
Client
2
A1
3
A2 A3 A4
Client contacts NameNode to write data
NameNode says write it to these nodes
Client sequentially
writes blocks to DataNode

How HDFS Works - Writes
NameNodeClient
A1 A2 A3 A4 A1A1 A2A2
A3A3A4 A4
DataNodes replicate data
blocks, orchestrated
by the NameNode

How HDFS Works - Reads
NameNodeClient
A1 A2 A3 A4 A1A1 A2A2
A3A3A4 A4
1
2
3
Client contacts NameNode to read data
NameNode says you can find it here
Client sequentially
reads blocks from DataNode

Hadoop MapReduce 1.x
Moves the code to the data
JobTracker
– Master service to monitor jobs
TaskTracker
– Multiple services to run tasks
– Same physical machine as a DataNode
A job contains many tasks
A task contains one or more task attempts

How MapReduce Works
DataNode A
A1 A2 A4 A2 A1 A3 A3 A2 A4 A4 A1 A3
JobTracker
1
Client
4
2
B1 B3 B4 B2 B3 B1 B3 B2 B4 B4 B1 B2
3
DataNode B DataNode C DataNode D
TaskTracker A TaskTracker B TaskTracker C TaskTracker D
Client submits job to JobTracker
JobTracker submits
tasks to TaskTrackers
Job output is written to
DataNodes w/replication
JobTracker reports metrics

MapReduce Paradigm
Data processing system with two key phases
Map
– Perform a map function on key/value pairs
Reduce
– Perform a reduce function on key/value groups
Groups created by sorting map output

Reduce Task 0 Reduce Task 1
Map Task 0 Map Task 1 Map Task 2
(0, "hadoop is fun") (52, "I love hadoop") (104, "Pig is more fun")
("hadoop", 1)
("is", 1)
("fun", 1)
("I", 1)
("love", 1)
("hadoop", 1)
("Pig", 1)
("is", 1)
("more", 1)
("fun", 1)
("hadoop", {1,1})
("is", {1,1})
("fun", {1,1})
("love", {1})
("I", {1})
("Pig", {1})
("more", {1})
("hadoop", 2)
("fun", 2)
("love", 1)
("I", 1)
("is", 2)
("Pig", 1)
("more", 1)
SHUFFLE AND SORT
Map Input
Map Output
Reducer Input Groups
Reducer Output

Word Count
Count the number of times
each word is used in a body
of text
Map input is a line of text
Reduce output a word and
the count
map(byte_offset, line)
foreach word in line
emit(word, 1)
reduce(word, counts)
sum = 0
foreach count in counts
sum += count
emit(word, sum)

Mapper Code
public class WordMapper
extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable ONE = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, ONE);
}
}
}

Reducer Code
public class IntSumReducer
extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values,
Context context) {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}

Pivotal HD

Pivotal HD
World‟s first true SQL processing for enterprise-ready
Hadoop
100% Apache Hadoop-based platform
Virtualization and cloud ready with VMWare and Isilon

Pivotal HD Architecture
HDFS
HBase
Pig, Hive, Mah
out
MapReduce
Sqoop Flume
Resource
Management
& Workflow
Yarn
ZooKeeper
Deploy,
Configure,
Monitor, Manage
Command
Center
Hadoop Virtualization (HVE)
Data Loader
Pivotal HD
Enterprise
Apache Pivotal HD Enterprise HAWQ
Xtension
Framework
Catalog
Services
Query
Optimizer
Dynamic Pipelining
ANSI SQL + Analytics
HAWQ – Advanced
Database Services
Spring

HAWQ

HAWQ: The Crown Jewel of Greenplum
 SQL compliant
 World-class query optimizer
 Interactive query
 Horizontal scalability
 Robust data management
 Common Hadoop formats
 Deep analytics

HAWQ
Query Processing
– Interactive and true ANSI
SQL support
– Multi-petabyte horizontal
scalability
– Cost-based parallel query
optimizer
– Programmable analytics
Database Services and
Management
– Scatter-gather data loading
– Row and column storage
– Workload management
– Multi-level partitioning
– 3rd-party tool & open client
interfaces

10+ Years MPP Database R&D to Hadoop
PRODUCT
FEATURES
CLIENT ACCESS
& TOOLS
Multi-Level Fault Tolerance
Shared-Nothing MPP
Parallel Query Optimizer
Polymorphic Data Storage™
CLIENT ACCESS
ODBC, JDBC, OLEDB,
MapReduce, etc.
MPP
ARCHITECTURE
Parallel Dataflow Engine
Software Interconnect
Scatter/Gather Streaming™ Data Loading
Online System Expansion Workload Management
ADAPTIVE
SERVICES
LOADING & EXT. ACCESS
Petabyte-Scale Loading
Trickle Micro-Batching
Anywhere Data Access
STORAGE & DATA ACCESS
Hybrid Storage & Execution
(Row- & Column-Oriented)
In-Database Compression
Multi-Level Partitioning
LANGUAGE SUPPORT
Comprehensive SQL
SQL 92, 99, 2003
OLAP Extensions
Analytics Extensions
3rd PARTY TOOLS
BI Tools, ETL Tools
Data Mining, etc
ADMIN TOOLS
Command Center
Package Manager

Query Optimizer
Physical plan contains
scans, joins, sorts, aggregations,
etc.
Cost-based optimization looks for
the most efficient plan
Global planning avoids sub-
optimal “SQL pushing” to
segments
Directly inserts “motion” nodes
for inter-segment communication
Execution Plan
ScanBars
b
HashJoinb.name =s.bar
ScanSells
s
Filterb.city ='SanFrancisco'
Projects.beer, s.price
MotionGather
MotionRedist(b.name)

Dynamic PipeliningTM
A supercomputing-based “soft-switch”
Core execution technology, borrowed from GPDB, allows us to run
complex job without materializing intermediate results.
Efficiently pumping streams of data between motion nodes during
query-plan execution
Delivers messages, moves data, collects results, and coordinates work
among the segments in the system
Dynamic PipeliningTM

Xtension Framework
Enables Intelligent query integration
with filter pushdown to
HBase, Hive, and HDFS
Supports common data formats such
Avro, Protocol Buffers and Sequence
Files
Provides extensible framework for
connectivity to other data sourcesHDFS HBase Hive
Xtension Framework

HAWQ Deployment
Dynamic Pipelining
... ...
......
Master
Servers & Name
Nodes
Query planning & dispatch
Segment
Servers & Data
Nodes
Query processing &
data storage
External
Sources
Loading, streami
ng, etc.
HDFS
ODBC/JDBC Driver

How HAWQ Works

Spring for Apache
Hadoop
Simplify developing Hadoop Applications

Developer observations on Hadoop
Hadoop has a poor out of the box programming model
Non trivial applications often become a collection of scripts
calling Hadoop command line applications
Spring aims to simplify developer Hadoop applications
– Leverage several Spring eco-system projects

Spring For Apache Hadoop - Features
Consistent programming and declarative configuration model
– Create, configure, and parameterize Hadoop connectivity and all job types
– Environment profiles – easily move application from dev to qa to production
Developer productivity
– Create well-formed applications, not spaghetti script applications
– Simplify HDFS access and FsShell API with support for JVM scripting
– Runner classes for MR/Pig/Hive/Cascading for small workflows
– Helper “Template” classes for Pig/Hive/HBase

Spring For Apache Hadoop – Use Cases
Apply across a wide range of use cases
– Ingest: Events/JDBC/NoSQL/Files to HDFS
– Orchestrate: Hadoop Jobs
– Export: HDFS to JDBC/NoSQL
Spring Integration and Spring Batch make this possible

• Standard Hadoop APIs
Counting Words – Configuring M/R
Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
Job.setJarByClass(WordCountMapper.class);
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);

Configuring Hadoop with Spring
<context:property-placeholder location="hadoop-dev.properties"/>
<hdp:configuration>
fs.default.name=${hd.fs}
</hdp:configuration>
<hdp:job id="word-count-job"
input-path=“${input.path}"
output-path="${output.path}“
jar=“hadoop-examples.jar”
mapper="examples.WordCount.WordMapper“
reducer="examples.WordCount.IntSumReducer"/>
<hdp:job-runner id=“runner” job-ref="word-count-job“
run-at-startup=“true“ />
input.path=/wc/input/
output.path=/wc/word/
hd.fs=hdfs://localhost:9000
applicationContext.xml
hadoop-dev.properties
Automatically determines
Output key and class

Injecting Jobs
Use DI to obtain reference to Hadoop Job
– Perform additional runtime configuration and submit
public class WordService {
@Autowired
private Job mapReduceJob;
public void processWords() {
mapReduceJob.submit();
}
}

input.path=/wc/input/
output.path=/wc/word/
hd.fs=hdfs://localhost:9000
Streaming Jobs and Environment Configuration
bin/hadoop jar hadoop-streaming.jar
–input /wc/input –output /wc/output
-mapper /bin/cat –reducer /bin/wc
-files stopwords.txt
<context:property-placeholder location="hadoop-${env}.properties"/>
<hdp:streaming id=“wc“ input-path=“${input}” output-path=“${output}”
mapper=“${cat}” reducer=“${wc}”
files=“classpath:stopwords.txt”>
</hdp:streaming>
env=dev java –jar SpringLauncher.jar applicationContext.xml
hadoop-dev.properties

Streaming Jobs and Environment Configuration
bin/hadoop jar hadoop-streaming.jar
–input /wc/input –output /wc/output
-mapper /bin/cat –reducer /bin/wc
-files stopwords.txt
<context:property-placeholder location="hadoop-${env}.properties"/>
<hdp:streaming id=“wc“ input-path=“${input}” output-path=“${output}”
mapper=“${cat}” reducer=“${wc}”
files=“classpath:stopwords.txt”>
</hdp:streaming>
env=qa java –jar SpringLauncher.jar applicationContext.xml
input.path=/gutenberg/input/
output.path=/gutenberg/word/
hd.fs=hdfs://darwin:9000
hadoop-qa.properties

• Access all “bin/hadoop fs” commands through
Spring‟s FsShell helper class
– mkdir, chmod, test
HDFS and Hadoop Shell as APIs
class MyScript {
@Autowired FsShell fsh;
@PostConstruct void init() {
String outputDir = "/data/output";
if (fsShell.test(outputDir)) {
fsShell.rmr(outputDir);
}
}
}

FsShell is designed to support JVM scripting languages
// use the shell (made available under variable fsh)
if (!fsh.test(inputDir)) {
fsh.mkdir(inputDir);
fsh.copyFromLocal(sourceFile, inputDir);
fsh.chmod(700, inputDir)
}
if (fsh.test(outputDir)) {
fsh.rmr(outputDir)
}
copy-files.groovy

Reference script and supply variables in application
configuration
<script id="setupScript" location="copy-files.groovy">
<property name="inputDir" value="${wordcount.input.path}"/>
<property name="outputDir" value="${wordcount.output.path}"/>
<property name=“sourceFile“ value="${localSourceFile}"/>
</script>
appCtx.xml

Small workflows
Often need the following steps
– Execute HDFS operations before job
– Run MapReduce Job
– Execute HDFS operations after job completes
Spring‟s JobRunner helper class sequences these steps
– Can reference multiple scripts with comma delimited names
<hdp:job-runner id="runner" run-at-startup="true"
pre-action="setupScript"
job="wordcountJob“
post-action=“tearDownScript"/>

Runner classes
Similar runner classes available for Hive and Pig
Implement JDK callable interface
Easy to schedule for simple needs using Spring
Can later „graduate‟ to use Spring Batch for more complex workflows
– Start simple and grow, reusing existing configuration
<hdp:job-runner id="runner“ run-at-startup=“false"
pre-action="setupScript“
job="wordcountJob“
post-action=“tearDownScript"/>
<task:scheduled-tasks>
<task:scheduled ref="runner" method="call" cron="3/30 * * * * ?"/>
</task:scheduled-tasks>

Spring‟s PigRunner
Execute a small Pig workflow
<pig-factory job-name=“analysis“ properties-location="pig-server.properties"/>
<script id="hdfsScript” location="copy-files.groovy">
<property name=“sourceFile" value="${localSourceFile}"/>
<property name="inputDir" value="${inputDir}"/>
<property name="outputDir" value="${outputDir}"/>
</script>
<pig-runner id="pigRunner“ pre-action="hdfsScript” run-at-startup="true">
<script location=“wordCount.pig">
<arguments>
inputDir=${inputDir}
outputDir=${outputDir}
</arguments>
</script>
</pig-runner>

PigTemplate - Configuration
Helper class that simplifies the programmatic use of Pig
– Common tasks are one-liners
Similar template helper classes for Hive and HBase
<pig-factory id="pigFactory“ properties-location="pig-server.properties"/>
<pig-template pig-factory-ref="pigFactory"/>

PigTemplate – Programmatic Use
public class PigPasswordRepository implements PasswordRepository {
@Autowired
private PigTemplate pigTemplate;
@Autowired
private String outputDir;
private String pigScript = "classpath:password-analysis.pig";
public void processPasswordFile(String inputFile) {
Properties scriptParameters = new Properties();
scriptParameters.put("inputDir", inputFile);
scriptParameters.put("outputDir", outputDir);
pigTemplate.executeScript(pigScript, scriptParameters);
}
}

Big Data problems are also integration problems
Collect Transform RT Analysis Ingest Batch Analysis Distribute Use
Spring Integration & Data
Spring Hadoop +
Batch
Spring MVCTwitter Search
& Gardenhose
Redis
Gemfire (CQ)

Spring Integration
 Implementation of Enterprise Integration Patterns
– Mature, since 2007
– Apache 2.0 License
 Separates integration concerns from processing logic
– Framework handles message reception and method invocation
• e.g. Polling vs. Event-driven
– Endpoints written as POJOs
• Increases testability
Endpoint Endpoint

Pipes and Filters Architecture
Endpoints are connected through Channels and exchange
Messages
$> cat foo.txt | grep the | while read l; do echo $l ; done
Endpoint Endpoint
Channel
Producer ConsumerFile RouteJMS TCP

Spring Batch
Framework for batch processing
– Basis for JSR-352
Born out of collaboration with
Accenture in 2007
Features
– parsers, mappers, readers, writers
– automatic retries after failure
– periodic commits
– synchronous and asynch processing
– parallel processing
– partial processing (skipping records)
– non-sequential processing
– job tracking and restart

Spring Integration and Batch for Hadoop
Ingest/Export
Event Streams – Spring Integration
– Examples
▪ Consume syslog events, transform and write to HDFS
▪ Consume twitter search results and write to HDFS
Batch – Spring Batch
– Examples
▪ Read log files on local file system, transform and write to HDFS
▪ Read from HDFS, transform and write to JDBC, HBase, MongoDB,…

Spring Data, Integration, & Batch for Analytics
Realtime Analytics – Spring Integration & Data
– Examples – Service Activator that
▪ Increments counters in Redis or MongoDB using Spring Data helper libraries
▪ Create Gemfire Continuous Queries using Spring Gemfire
Batch Analytics – Spring Batch
– Orchestrate Hadoop based workflows with Spring Batch
– Also orchestrate non-hadoop based workflows

Ingesting – Syslog into HDFS
Use SI‟s syslog adapter
Perform transformation on data
Route to specific channels based
on category
One route leads to HDFS and
filtered data stored in Redis

Ingesting – Multi-node syslog into HDFS
Syslog collection across multiple
machines
Break processing chain at
channel boundaries
Use SI‟s TCP adapters to forward
events
– Or other SI middleware adapters

Hadoop Analytical workflow managed by
Spring Batch
 Reuse same Batch infrastructure
and knowledge to manage
Hadoop workflows
 Step can be any Hadoop job
type or HDFS script

Spring Batch Configuration for Hadoop
<job id="job1">
<step id="import" next="wordcount">
<tasklet ref=“import-tasklet"/>
</step>
<step id=“wc" next="pig">
<tasklet ref="wordcount-tasklet"/>
</step>
<step id="pig">
<tasklet ref="pig-tasklet“></step>
<split id="parallel" next="hdfs">
<flow><step id="mrStep">
<tasklet ref="mr-tasklet"/>
</step></flow>
<flow><step id="hive">
<tasklet ref="hive-tasklet"/>
</step></flow>
</split>
<step id="hdfs">
<tasklet ref="hdfs-tasklet"/></step>
</job>

• Use Spring Batch‟s
– MutliFileItemReader
– JdbcItemWriter
Exporting HDFS to JDBC
<step id="step1">
<tasklet>
<chunk reader=“flatFileItemReader" processor="itemProcessor" writer=“jdbcItemWriter"
commit-interval="100" retry-limit="3"/>
</chunk>
</tasklet>
</step>

Relationship between Spring Projects

Next Steps – Spring XD
New open source umbrella project to support common big
data use cases
– High throughput distributed data ingestion into HDFS
▪ From a variety of input sources
– Real-time analytics at ingestion time
▪ Gathering metrics, counting values, Gemfire CQ…
– On and off Hadoop workflow orchestration
– High throughput data export
▪ From HDFS to a RDBMS or NoSQL database.
XD = eXtreme Data or y= mx + b

Next Steps – Spring XD
Consistent model that spans the 4 use-case categories
Move beyond delivering a set of libraries
– Provide out-of-the-box executable servier
– High level DSL to configure flows and jobs
▪ http | hdfs
– Pluggable module system
See blog post for more information
– Github: http://github.com/springsource/spring-xd
Get involved!

Resources
Pivotal
– goPivotal.com
Spring Data
– http://www.springsource.org/spring-data
– http://www.springsource.org/spring-hadoop
Spring Data Book - http://bit.ly/sd-book
– Part III on Big Data
Example Code https://github.com/SpringSource/spring-data-book
Spring XD http://github.com/springsource/spring-xd

Pivotal HD and Spring for Apache Hadoop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Pivotal HD and Spring for Apache Hadoop

Similar to Pivotal HD and Spring for Apache Hadoop (20)

Recently uploaded

Recently uploaded (20)

Pivotal HD and Spring for Apache Hadoop

Editor's Notes