The document discusses data storage and processing. Data can be stored in memory or on disk using file systems like local XFS/ZFS or distributed systems like HDFS, S3, or Ceph. Distributed file systems allow for parallel processing of data by moving computation to the data locations. This map-reduce framework involves mapping functions to distributed data segments followed by reducing the results. Hadoop uses HDFS for storage and the MapReduce framework for distributed computation on large datasets across clusters.

1. Storage + Compute

2. Data Processing

3. Data Processing
1. Where is data stored ?

4. Data Processing
1. Where is data stored ?
2. Where does the compute run ?

5. Storage

6. Storage
● In Memory

7. Storage
● In Memory
● On Disk

8. Storage
● In Memory
● On Disk
○ File System

9. Storage
● In Memory
● On Disk
○ File System
■ Local file system - xfs , zfs , etc

10. Storage
● In Memory
● On Disk
○ File System
■ Local file system - xfs , zfs
■ Distributed File System

11. Storage
● In Memory
● On Disk
○ File System
■ Local file system - xfs , zfs
■ Distributed File System
● HDFS - Hadoop Distributed File System
● S3
● Ceph etc

12. H-Distributed-FS

13. H-Distributed-FS
Motivation:

14. H-Distributed-FS
Motivation:
● Parallel Processing of Data

15. H-Distributed-FS
Motivation:
● Parallel Processing of Data
○ When data is distributed, it can be processed in parallel *
* some problem statements are not a fit for this.

16. H-Distributed-FS
Motivation:
● Parallel Processing of Data
○ When data is distributed, it can be processed in parallel *
● Computation goes to data and not data to computation
* some problem statements are not a fit for this.

17. Word Count Problem
● Single file

18. Word Count Problem
● Single file
○ O(n) time
complexity

19. Word Count Problem
● Distributed data

20. Word Count Problem
● Distributed data
O(m) time complexity
‘m’ = size of largest file

21. Parallel Compute

22. Parallel Compute
● Parallel Computation on the data

23. Parallel Compute
● Parallel Computation on the data
● Computation goes to data & not data
to computation. Compute works over
the data local to it.

24. Parallel Compute
● Parallel Computation on the data
● Computation goes to data & not data
to computation. Compute works over
the data local to it.
● A final aggregation happens.

25. Parallel Compute
● Parallel Computation on the data
● Computation goes to data & not data
to computation. Compute works over
the data local to it.
● A final aggregation happens.
This Compute Paradigm is called
“MAP REDUCE framework”

26. Storage + Compute

27. Hdfs + MapReduce

28. Hdfs + MapReduce = Hadoop

29. Some Notes
● Hadoop Version 1 - has Map Reduce (MR) framework
● Hadoop Version 2 - has Yarn (resource scheduler)
● Apache Spark
○ Alternative to Map Reduce Compute framework
○ Can use Apache Spark with data in HDFS
○ Can run in Yarn with data in s3 ! (we just use yarn features
without using MR and HDFS features) :)

30. Hdfs + MapReduce = Hadoop
By Vishnu Rao
mash213.wordpress.com
linkedin.com/in/213vishnu