The document summarizes the Hadoop stack and its components for storing and analyzing big data. It includes file storage with HDFS, data processing with MapReduce, data access tools like Hive and Pig, and security/monitoring with Kerberos and Nagios. HDFS uses metadata to track file locations across data nodes in a fault-tolerant manner similar to a file system.

1. Big Data Summit
Dave Wolcott
1

2. High Level
2

3. Hadoop Stack
3
Extract Load Transform
HDFS
Propagate
RDBMS/Files
API Access
Business Intelligence
Staging
ConsumableData
Security
Kerberos
Data Chronology
Hive/Pig
Hbase
Meta Data
HCatalog
Job Scheduling
oozie
Extract to RDBMS
sqoop
Monitoring Tools
Nagios, Ganglia, Ambari
Direct Access to Raw Data
Hue
Data Serialization
Avro
Governance
HadoopStackandDataAccess
Data Extraction
Flume
Google Analytics
Data Movement
Map Reduce
lyndaLogs
User Sessions
ServicesandAPI`s

4. 4
HDFS Analogy
File metadata
/user/dave/data -> 1,2,3
/user/subash/data -> 4,5
3 1
5
2
3
5
1
5
2
4
1 4
3
2
Name Node
Data Node

5. 5
File Meta Data:
File Name
File Type
Access Rights
Owners
Timestamps
Size
Data block Pointers
1
2
3
4
5
6
7
8
9
10
11
12
Data Block
Data Block
Data Block
Data Block
Data Block
Data Block
Data Block
Data Block
Data Block
Data Block
Indirect
HDFS Analogy

6. 6
File Meta Data:
File Name
File Type
Access Rights
Owners
Timestamps
Size
Data block Pointers
Data Block
Data Block
Data Block
Data Block
Data Block
Data Block
Data Block
Data Block
Data Block
Data Block
HDFS Analogy

7. 7
X
X
X
X
X
X
X
X
HDFS Analogy

8. 8
X
X
X
X
X
X
X
X
HDFS Analogy

9. 9
HDFS Analogy
File metadata
/user/dave/data -> 1,2,3
/user/subash/data -> 4,5
3 1
5
2
3
5
1
5
2
4
1 4
3
2
Name Node
Data Node

10. Toronto, 20
Whitby, 25
New York, 22
Rome, 32
Toronto, 4
Rome, 33
New York, 18
New York, 18
Toronto, 18
Whitby, 27
New York, 32
Rome, 37
Toronto, 32
Whitby, 20
New York, 33
Rome, 38
Toronto, 22
Whitby, 19
Rome, 32
Whitby, 22
New York, 20
Rome, 31
New York, 17
Toronto, 31
Whitby, 22
New York, 19
Toronto, 28
Rome, 30
New York, 32
Toronto, 18
Whitby, 23
New York, 32
Rome, 37
Toronto, 22
Whitby, 20
New York, 31
Toronto, 32
Whitby, 27
New York, 33
Rome, 38
Toronto, 20
Whitby, 25
New York, 22
Rome, 33
Map Reduce
FiveParallelProcesses
ReducetheFivetoOne
Toronto, 32
Whitby, 27
New York, 33
Rome, 37
Toronto, 22
Whitby, 22
New York, 20
Rome, 38
Toronto, 31
Whitby, 22
New York, 19
Rome, 31
Toronto, 22
Whitby, 23
New York, 32
Rome, 37