Hadoop is a tool for solving big data problems involving large amounts of data by distributing storage and processing across clusters of commodity hardware. It allows processing of large datasets in parallel by breaking jobs into smaller tasks, running them on multiple nodes, and combining the results. It also replicates data across multiple nodes to prevent data loss from hardware failures. The key components of Hadoop are HDFS for distributed storage, MapReduce for distributed processing, and additional services like YARN and Hive for data processing frameworks.

1. HADOOP 101
A really quick overview of what it is
(and why you should care...)
Monday, 30 September, 13

2. HADOOP IS A
TOOLTO SOLVE
BIG DATA PROBLEMS
Monday, 30 September, 13

3. HADOOP IS A
(ONE KIND OF)
TOOLTO SOLVE
(SOME KINDS OF)
BIG DATA PROBLEMS
Monday, 30 September, 13

4. OK....
SO WHAT IS A
BIG DATA PROBLEM?
Monday, 30 September, 13

5. “A PROBLEM IS A
BIG DATA PROBLEM
WHENTHE SIZE OFTHE DATA
IS PART OFTHE PROBLEM”
Monday, 30 September, 13

6. A fewTerabytes of Data...
Monday, 30 September, 13

7. Monday, 30 September, 13

8. Monday, 30 September, 13

9. Text processing--a few hours?
Monday, 30 September, 13

10. But what if you have more data?
Monday, 30 September, 13

11. Network Storage--Petabytes!
Monday, 30 September, 13

12. Network Storage--Petabytes!
Monday, 30 September, 13

13. What if you need compute power for complex algorithms?
Monday, 30 September, 13

14. 8 core? 16 Cores? 64 cores? 512 GB RAM?
Monday, 30 September, 13

15. More processing in one box--price grows exponentially
0
75,000
150,000
225,000
300,000
16 32 48 64 80 96 112 128
Single Machine
Monday, 30 September, 13

16. Compute cost for many commodity machines scales linearly
0
75,000
150,000
225,000
300,000
16 32 48 64 80 96 112 128
Single Machine Many Commodity Maciines
Monday, 30 September, 13

17. A network of commodity computers
Monday, 30 September, 13

18. Run jobs on part of the data on each one. Compile the results.
Monday, 30 September, 13

19. Let’s add a computer to manage the process of
job delegation, merging the results...
Monday, 30 September, 13

20. We also need something to keep track of what files are where, so
we know what data needs to be computed...
Monday, 30 September, 13

21. When you have a lot of computers, and even more hard drives,
one thing I can guarantee...
Monday, 30 September, 13

22. Computers will eventually fail.
Monday, 30 September, 13

23. Computers will eventually fail.
Monday, 30 September, 13

24. Hard drives will eventually fail.
Monday, 30 September, 13

25. Hard drives will eventually fail.
Monday, 30 September, 13

26. Hard drives will eventually fail.
Monday, 30 September, 13

27. Hard drives will eventually fail.
Monday, 30 September, 13

28. Even whole racks will fail.
Monday, 30 September, 13

29. If a computer fails and you only have one copy of your data...
Monday, 30 September, 13

30. You will be very, very unhappy.
Monday, 30 September, 13

31. So lets store multiple copies of the data. Hard drives are CHEAP!
Monday, 30 September, 13

32. So lets store multiple copies of the data. Hard drives are CHEAP!
Monday, 30 September, 13

33. So lets store multiple copies of the data. Hard drives are CHEAP!
Monday, 30 September, 13

34. So lets store multiple copies of the data. Hard drives are CHEAP!
Monday, 30 September, 13

35. If one hard drive fails... we are still OK
Monday, 30 September, 13

36. If one computer fails... we are still OK
Monday, 30 September, 13

37. Even if a whole rack fails... we are still OK
Monday, 30 September, 13

38. Once we find a failure let’s have the system recopy the copies.
Monday, 30 September, 13

39. Guess, what? We’ve just invented Hadoop!
Monday, 30 September, 13

40. So let’s talk about the pieces of Hadoop.
Monday, 30 September, 13

41. Data nodes store and manage the data on a single “slave” computer
Data Node
Monday, 30 September, 13

42. Task trackers manage the compute
Data Node
Task Tracker
Monday, 30 September, 13

43. Job tracker manages task trackers, ships code to compute nodes
Data Node
Task Tracker
Job Tracker
Monday, 30 September, 13

44. Name node manages distribution and replication on the data nodes
Data Node
Task Tracker
Job Tracker
Name Node
Monday, 30 September, 13

45. Map Reduce
Task Tracker
Job Tracker
Monday, 30 September, 13

46. HDFS (Hadoop Distributed File System)
Data Node
Name Node
Monday, 30 September, 13

47. VISUAL EXAMPLE
Monday, 30 September, 13

48. MAP
Monday, 30 September, 13

49. SHUFFLE
Monday, 30 September, 13

50. REDUCE
Monday, 30 September, 13

51. PUTTING IT ALLTOGETHER
Monday, 30 September, 13

52. SO WHAT?
Monday, 30 September, 13

53. THAT’S WHAT!
Monday, 30 September, 13

54. OPPORTUNITY!
Monday, 30 September, 13

55. HADOOP ECOSYSTEM
Monday, 30 September, 13