Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of computers. It was created in 2006 by Doug Cutting and is based on Google's paper describing its Google File System and MapReduce. Hadoop allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

1. A Hadoop Primer
Feb 2011
10.20.2005

2. http://redmonk.com/public/hadoop.pdf
2

3. The Background
3

4. October, 2003
4

5. December, 2004
5

6. Map::Reduce
6

7. Job::Map Reduce::Output
7

8. Counting Shakespeare
8

9. The Birth of Hadoop
9

10. 10

11. 11

12. Project Architecture
Source: Running Hadoop On Ubuntu Linux, Michael G. Noll, 8.8.07
12

13. Project Traction
13

14. Employment Potential
14

15. Hadoop Users
15

16. Why Hadoop?
16

17. More Machines = More Faster
17

18. The reason everyone knows
18

19. BIG DATA
19

20. “The big issue is not that everyone will
suddenly operate at petabyte scale; a lot of
folks do not have that much data.
The more important topics are the specifics
of the storage and processing infrastructure
and what approaches best suit each
problem.”
- Bradford Cross, Flightcaster/Woven
20

21. The reason not everyone
knows
21

22. ru d
U st tu e Data
n r
c
22

23. What Hadoop Is
23

24. “build Amazon's product search indices”
“build the recommender system for behavioral targeting”
“ETL style processing and statistics generation”
“information extraction & search”
“searching and analysis of millions of rental bookings”
“we use Hadoop to summarize of user's tracking data”
“we use Hadoop to store ad serving logs”
“the freedom to query the data in an ad-hoc manner”
“generating web graphs on 100 nodes”
“we use Hadoop for batch-processing large RDF datasets”
“facial similarity and recognition across large datasets“
“We are using Hadoop and Nutch to crawl Blog posts”
“Used for ETL & data analysis on terascale datasets”
Source: http://wiki.apache.org/hadoop/PoweredBy
24

25. What Hadoop Isn't
25

26. A relational database killer
No Yes
26

27. Beyond Hadoop
27

28. The Hadoop Ecosystem
28

29. What We Use Hadoop For
29

30. Crawling Largeish
Unstructured Datasets
30

31. Like 1.3M StackOverflow Questions
31

32. Or 1.7M HackerNews Entries
32

33. Or Years of Apache Log Files
33

34. How to Get Started
34

35. We use Cloudera
35

36. Mostly because it's easy
36

37. This easy
37

38. Or if you prefer
38

39. Or maybe this
39

40. QUESTIONS
40

41. Student? Talk to us
41