This document introduces Hadoop, an open-source software platform for distributed storage and processing of large datasets across clusters of computers. It discusses the key components of Hadoop including the Hadoop Distributed File System (HDFS) for storage, MapReduce for processing, and YARN for resource management. The document also briefly introduces related projects like Pig for data flows and Hive for SQL-like queries that build on Hadoop.

1. INTRODUCTION TO HADOOP
Brest – 29 octobre 2014
David Morin - @davAtBzh

2. Me
David Morin
@davAtBzh
Solutions Engineer at

3. 3
What is Hadoop ?

4. 4
An elephant – This one ?

5. 5
No, this one !

6. 6
The father

7. 7
Let's go !

8. 8
Let's go !

9. 9
Timeline

10. 10
Hadoop fundamentals
● Distributed FileSystem for high volume of data
● Use of common servers (limit costs)
● Scalable / fault tolerance

11. 11
Hadoop Distributed FileSystem

12. 12
Hadoop Distributed FileSystem

13. 13
Mapreduce

14. 14
Mapreduce : word count
Map Reduce

15. 15
Data Locality Optimization

16. 16
Mapreduce in action

17. 17
Hadoop v1 : drawbacks
– One Namenode : SPOF
– One Jobtracker : SPOF and un-scalable (nodes
limitation)
– MapReduce only : open this platform to non MR
applications

18. 18
Hadoop v2
Improvements :
– HDFS v2 : Secondary namenode
– YARN (Yet Another Resource Negociator)
● JobTracker => Resource Manager + Applications
Master (more than one)
● Can be used by non MapReduce applications
– MapReduce v2 : uses Yarn

19. 19
Hadoop v2

20. 20
YARN

21. 21
YARN

22. 22
YARN

23. 23
YARN

24. 24
YARN

25. 25
YARN

26. 26

27. 27
Pig
● With Pig write MR Jobs becomes easy
● Dataflow model : data is the key !
● Langage : PigLatin
● No limit : Used Defined Functions
http://pig.apache.org/docs/r0.13.0/

28. 28
● Pig-Wordcount
Pig
lines = LOAD '/user/XXX/file.txt' AS (line:chararray);
words = FOREACH lines GENERATE FLATTEN(TOKENIZE(line)) AS word;
grouped = GROUP words BY word;
wordcount = FOREACH grouped GENERATE group, COUNT(words);
DUMP wordcount;

29. 29
Import …
public class WordCount2 {
Pig
public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{
static enum CountersEnum { INPUT_WORDS }
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
private boolean caseSensitive;
private Set<String> patternsToSkip = new HashSet<String>();
private Configuration conf;
private BufferedReader fis;
...
=> 130 lines of code !

30. 30
● SQL like : HQL
● UDFs
● Hive-Wordcount
Hive
CREATE TABLE docs (line STRING);
LOAD DATA INPATH 'text' OVERWRITE INTO TABLE docs;
CREATE TABLE word_counts AS
SELECT word, count(1) AS count FROM
(SELECT explode(split(line, 's')) AS word FROM docs) w
GROUP BY word
ORDER BY word;

31. 31
Zookeeper
● Distributed coordination service
● Dynamic configuration
● Distributed locking

32. 32
Batch but not only..

33. 33
??