Talk held at codemotion Berlin 2013 on 10.05.2013 in Berlin, Germany

1. Introduction to the
Hadoop ecosystem

2. About me

3. About us

4. Why Hadoop?

5. Why Hadoop?

6. Why Hadoop?

7. Why Hadoop?

8. Why Hadoop?

9. Why Hadoop?

10. Why Hadoop?

11. How to scale data?
w1 w2 w3
r1 r2 r3

12. But…

13. But…

14. What is Hadoop?

15. What is Hadoop?

16. What is Hadoop?

17. What is Hadoop?

18. The Hadoop App Store
HDFS MapRed HCat Pig Hive HBase Ambari Avro Cassandra
Chukwa
Intel
Sync
Flume Hana HyperT Impala Mahout Nutch Oozie Scoop
Scribe Tez Vertica Whirr ZooKee Cloudera Horton MapR EMC
IBM Talend TeraData Pivotal Informat Microsoft. Pentaho Jasper
Kognitio Tableau Splunk Platfora Rack Karma Actuate MicStrat

19. Data Storage

20. Data Storage

21. Hadoop Distributed File System
•
•
•

22. Hadoop Distributed File System
•
•

23. HDFS Architecture

24. Data Processing

25. Data Processing

26. MapReduce
•
•
•

27. Typical large-data problem
•
•
•
•
•

28. MapReduce Flow
𝐤 𝟏 𝐯 𝟏 𝐤 𝟐 𝐯 𝟐 𝐤 𝟒 𝐯 𝟒 𝐤 𝟓 𝐯 𝟓 𝐤 𝟔 𝐯 𝟔𝐤 𝟑 𝐯 𝟑
a 𝟏 b 2 c 9 a 3 c 2 b 7 c 8
a 𝟏 b 2 c 3 c 6 a 3 c 2 b 7 c 8
a 1 3 b 𝟐 7 c 2 8 9
a 4 b 9 c 19

29. Jobs & Tasks
•
•
•
•

30. Combined Hadoop Architecture

31. Word Count Mapper in Java
public class WordCountMapper extends MapReduceBase implements
Mapper<LongWritable, Text, Text, IntWritable>
{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text,
IntWritable> output, Reporter reporter) throws IOException
{
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens())
{
word.set(tokenizer.nextToken());
output.collect(word, one);
}
}
}

32. Word Count Reducer in Java
public class WordCountReducer extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable>
{
public void reduce(Text key, Iterator values, OutputCollector
output, Reporter reporter) throws IOException
{
int sum = 0;
while (values.hasNext())
{
IntWritable value = (IntWritable) values.next();
sum += value.get();
}
output.collect(key, new IntWritable(sum));
}
}

33. Scripting for Hadoop

34. Scripting for Hadoop

35. Apache Pig
•
•
•
•

36. Pig in the Hadoop ecosystem
Hadoop Distributed File System
Distributed Programming Framework
Metadata Management
Scripting

37. Pig Latin
users = LOAD 'users.txt' USING PigStorage(',') AS (name,
age);
pages = LOAD 'pages.txt' USING PigStorage(',') AS (user,
url);
filteredUsers = FILTER users BY age >= 18 and age <=50;
joinResult = JOIN filteredUsers BY name, pages by user;
grouped = GROUP joinResult BY url;
summed = FOREACH grouped GENERATE group,
COUNT(joinResult) as clicks;
sorted = ORDER summed BY clicks desc;
top10 = LIMIT sorted 10;
STORE top10 INTO 'top10sites';

38. Pig Execution Plan

39. Try that with Java…

40. SQL for Hadoop

41. SQL for Hadoop

42. Apache Hive
•
•

43. Hive in the Hadoop ecosystem
Hadoop Distributed File System
Distributed Programming Framework
Metadata Management
Scripting Query

44. Hive Architecture

45. Hive Example
CREATE TABLE users(name STRING, age INT);
CREATE TABLE pages(user STRING, url STRING);
LOAD DATA INPATH '/user/sandbox/users.txt' INTO
TABLE 'users';
LOAD DATA INPATH '/user/sandbox/pages.txt' INTO
TABLE 'pages';
SELECT pages.url, count(*) AS clicks FROM users JOIN
pages ON (users.name = pages.user)
WHERE users.age >= 18 AND users.age <= 50
GROUP BY pages.url
SORT BY clicks DESC
LIMIT 10;

46. Bringing it all together…

47. Online Advertising

48. Getting started…

49. Hortonworks Sandbox

50. Hadoop Training
•
•
•
•
•
•
•
•
•

51. The end…or the beginning?