Hadoop is an open source software framework that allows for the distributed storage and processing of extremely large datasets across clusters of commodity hardware. It uses a scalable distributed file system called HDFS to store data reliably, and its MapReduce programming model enables parallel processing of huge datasets across large clusters of servers. The Hadoop ecosystem includes additional popular tools like Pig, Hive, HBase, and Zookeeper that provide SQL-like querying, real-time database access, and coordination services to make the Hadoop platform more full-featured and user-friendly.

1. Hadoop
Chris Sharkey
today
@shark2900

4. Why use hadoop?
• Uses commodity hardware
• Stores petabytes of data reliably
• Allows for huge distributed computations
• Open source project and ecosystem

5. Core concepts

6. Single Computer
Map Reduce
HDFS
Task Tracker
Data Node
or

7. Cluster
Task Tracker
Data Node
Job Tracker
Task Tracker
Data Node
Task Tracker
Data Node
Name Node

8. The ecosystem

9. Pig
• Programing language
• High level for MapReduce
• ‘Compiler for MapReduce’
HDFS
MapReduce
Pig

10. Hive
• SQL like interface
• Querying & RD functionality
• Familiar to traditional business
intelligence operations
HDFS
MapReduce
Hive

11. HBase
• NoSQL database
• Based off of HDFS
• Real time updating and access
HDFS
MapReduce
HBase

12. Zookeeper
• Coordination services for many
server architects. Distributed
application management with
added reliability
HDFS
MapReduce
HBase
Zookeeper

13. Combined System
HDFS
MapReduce
HBase
Zookeeper
Hive
Pig

15. Summary
• Powerful system for ‘big data’
• Commodity hardware
• Redundant and reliable
• Ecosystem affords modularity
• Ecosystem affords relevance
• Distributed analytics. Learn form petabytes of data

16. Questions?

17. Thank you