This document discusses Spring for Apache Hadoop, a framework that simplifies using Apache Hadoop and related projects like HBase and Hive within the Spring programming model. It provides wrappers and configuration for common Hadoop tasks like MapReduce jobs, scripting, and accessing Hadoop databases and data processing engines. The goals are to provide a programmatic model for the Hadoop ecosystem, simplify client libraries, and leverage Spring features. It supports various Hadoop distributions and provides interfaces for MapReduce, HBase, Hive, Pig and other Hadoop technologies.

1. Spring for Apache
Hadoop
By Zenyk Matchyshyn

2. Agenda
• Goals of the project
• Hadoop Map/Reduce
• Scripting
• HBase
• Hive
• Pig
• Other
• Alternatives
2

3. Big Data – Why?
Because of Terabytes and Petabytes:
• Smart meter analysis
• Genome processing
• Sentiment & social media analysis
• Network capacity trending & management
• Ad targeting
• Fraud detection
3

4. Goals
• Provide programmatic model to work with
Hadoop ecosystem
• Simplify client libraries usage
• Provide Spring friendly wrappers
• Enable real-world usage as a part of Spring
Batch & Spring Integration
• Leverage Spring features
4

5. Supported distros
• Apache Hadoop
• Cloudera CDH
• Greenplum HD
5

6. HADOOP
6

7. Hadoop
7
Hadoop
Map/Reduce
HDFS
HBase
Pig Hive

8. Hadoop basics
Split Map Shuffle Reduce
8
Dog ate the bone
Cat ate the fish
Dog, 1
Ate, 1
The, 1
Bone, 1
Cat, 1
Ate, 1
The, 1
Fish,1
Dog, 1
Ate, {1, 1}
The, {1, 1}
Bone, 1
Cat, 1
Fish,1
Dog, 1
Ate, 2
The, 2
Bone, 1
Cat, 1
Fish,1

9. Configuration
9
<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/hadoop"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:beans="http://www.springframework.org/schema/beans"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/hadoop
http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">
<context:property-placeholder location="hadoop.properties"/>
<configuration>
fs.default.name=${hd.fs}
mapred.job.tracker=${hd.jt}
</configuration>
………………….
</beans:beans

10. Job definition
10
<hdp:job id=“hadoopJob"
input-path="${wordcount.input.path}"
output-path="${wordcount.output.path}"
libs="file:${app.repo}/supporting-lib-*.jar"
mapper="org.company.Mapper"
reducer="org.company.Reducer"/>
Configuration conf = new Configuration();
Job job = new Job(conf, “hadoopJob");
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(Maper.class);
job.setReducerClass(Reducer.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new
Path(args[1]));
job.waitForCompletion(true);

11. Job Execution
11
<hdp:job-runner id="runner" run-at-startup="true"
pre-action=“someScript“
post-action=“someOtherScript“
job-ref=“hadoopJob" />

12. • Hadoop Streaming:
• Hadoop Tool Executor:
Other approaches
12
<hdp:streaming id="streaming"
input-path="/input/" output-path="/ouput/"
mapper="${path.cat}" reducer="${path.wc}"/>
<hdp:tool-runner id="someTool" tool-class="org.foo.SomeTool" run-at-startup="true">
<hdp:arg value="data/in.txt"/>
<hdp:arg value="data/out.txt"/>
property=value
</hdp:tool-runner>

13. SCRIPTING
13

14. Details
• Supports JVM languages from JSR-223
(Groovy, JRuby, Jython, Rhino)
• Exposes SimplerFileSystem
• Provides implicit variables
• Exposes FsShell to mimic HDFS shell
• Exposes DistCp to mimic distcp from Hadoop
14

15. Example
15
<hdp:script-tasklet id="script-tasklet">
<hdp:script language="groovy">
inputPath = "/user/gutenberg/input/word/"
outputPath = "/user/gutenberg/output/word/"
if (fsh.test(inputPath)) {
fsh.rmr(inputPath) }
if (fsh.test(outputPath)) {
fsh.rmr(outputPath) }
inputFile = "src/main/resources/data/nietzsche-chapter-1.txt"
fsh.put(inputFile, inputPath)
</hdp:script>
</hdp:script-tasklet>

16. HBASE
16

17. HBase
17
Hadoop
Map/Reduce
HDFS
HBase
Pig Hive

18. HBase basics
• Distributed, column oriented store
• Independent of Hadoop
• No translation into Map/Reduce
• Stores data in MapFiles (indexed SequenceFiles)
18
Create ‘sometable’, ‘clmnfamily1’
Put ‘sometable’, ‘row_id1’, ‘clmnfamily1:c1’, ‘some values’
Scan ‘sometable’

19. Features
• Easy connection interface
• Thread safe
• DAO friendly support and wrappers:
• HbaseTemplate
• TableCallback
• RowMapper
• ResultsExtractor
• Binding table to current thread
19

20. Example - beans
20
<hdp:hbase-configuration/>
<bean id="hbaseTemplate"
class="org.springframework.data.hadoop.hbase.HbaseTemplate"
p:configuration-ref="hbaseConfiguration"/>

21. Example - code
21
template.execute("MyTable", new TableCallback<Object>() {
@Override
public Object doInTable(HTable table) throws Throwable {
Put p = new Put(Bytes.toBytes("SomeRow"));
p.add(Bytes.toBytes("SomeColumn"), Bytes.toBytes("SomeQualifier"), Bytes.toBytes("AValue"));
table.put(p);
return null;
}
});
List<String> rows = template.find("MyTable", "SomeColumn", new RowMapper<String>() {
@Override
public String mapRow(Result result, int rowNum) throws Exception {
return result.toString();
}
}));

22. HIVE
22

23. Hive
23
Hadoop
Map/Reduce
HDFS
HBase
Pig Hive

24. Hive basics
• SQL-like interface - HiveQL
• Has its own structure
• Not a pipeline like Pig
• Basically a distributed data warehouse
• Has execution optimization
24

25. Features
• Hive server
• DAO friendly Hive Thrift Client simplification
• Hive JDBC driver within Spring DAO ecosystem
• Hive scripting
• Thread safe
25

26. Example - beans
26
<hdp:hive-server host=“hivehost" port="10001" />
<hdp:hive-template />
<hdp:hive-client-factory host="some-host" port="some-port" >
<hdp:script location="classpath:org/company/hive/script.q">
<arguments>ignore-case=true</arguments>
</hdp:script>
</hdp:hive-client-factory>
<hdp:hive-runner id="hiveRunner" run-at-startup="true">
<hdp:script>
DROP TABLE IF EXITS testHiveBatchTable;
CREATE TABLE testHiveBatchTable (key int, value string);
</hdp:script>
<hdp:script location="hive-scripts/script.q"/>
</hdp:hive-runner>

27. Example - template
27
return hiveTemplate.execute(new HiveClientCallback<List<String>>() {
@Override
public List<String> doInHive(HiveClient hiveClient) throws Exception {
return hiveClient.get_all_databases();
}
}));

28. PIG
28

29. Pig basics
29
Hadoop
Map/Reduce
HDFS
HBase
Pig Hive

30. Pig
• High level language for data analysis
• Uses PigLatin to describe data flows
(translates into MapReduce)
• Filters, Joins, Projections, Groupings, Counts,
etc.
• Example:
30
A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:float);
B = FOREACH A GENERATE name;
DUMP B;

31. Features
• Scripts execution
• DAO friendly template
• Thread safe
31

32. Example - beans
32
<hdp:pig-factory exec-type="LOCAL" job-name="pig-script" configuration-ref="hadoopConfiguration"
properties-location="pig-dev.properties”">
source=${pig.script.src}
<script location="org/company/pig/script.pig“/>
</hdp:pig-factory>
<hdp:pig-runner id="pigRunner" run-at-startup="true">
<hdp:script>
A = LOAD 'src/test/resources/logs/apache_access.log' USING PigStorage() AS (name:chararray, age:int);
B = FOREACH A GENERATE name;
DUMP B;
</hdp:script>
<hdp:script location="pig-scripts/script.pig">
<arguments>electric=sea</arguments>
</hdp:script>
</hdp:pig-runner>
<hdp:pig-template/>

33. Example - template
33
return pigTemplate.execute(new PigCallback<Set<String>() {
@Override
public Set<String> doInPig(PigServer pig) throws ExecException, IOException {
return pig.getAliasKeySet();
}
}));

34. Other features
• Cascading support
• Works well with Hadoop security
• Spring Batch tasklets
• Spring Integration support
34

35. Alternatives & related
• Apache Flume – distributed data collection
• Apache Oozie – workflow scheduler
• Apache Sqoop – SQL bulk import/export
35

36. Q/A
?
36