Apache Hadoop is an open source software framework created in 2005 for distributed storage and processing of large datasets across clusters of computers. It allows for reliable and scalable computation across large amounts of data and nodes. The core modules include Hadoop Common, HDFS for distributed file storage, YARN for job scheduling and cluster resource management, and MapReduce for distributed processing of large datasets. The Hadoop ecosystem has grown significantly and includes additional popular components such as Pig, Hive, HBase, Zookeeper, Oozie, Flume, Spark and Impala.

1. Big Data
Hadoop Stack

2. Lecture #1
Hadoop Beginnings

3. What is Hadoop?
Apache Hadoop is an open source
software framework for storage and
large scale processing of data-sets on
clusters of commodity hardware

4. Hadoop was created by Doug Cutting
and Mike Cafarella in 2005
Named the project after son's toy
elephant

5. Moving Computation to Data
Computation
Data

6. Scalability at Hadoop’s core!

7. Reliability!
Reliability!
Reliability!

8. Reliability!
Reliability!
Reliability!

9. Reliability!
Reliability!
Reliability!
Google File System

10. Once
a year
365 Computers
Once
a day

11. Hourly

12. New Approach to Data
Keep all data

13. New Kinds of Analysis
Schema-on read style

14. New Kinds of Analysis
Schema-on read styleNew analysis

15. Small Data & Complex
Algorithm
Large Data &
Simple Algorithm
Vs.

16. Lecture #2
Apache Framework
Hadoop Modules

17. Apache Framework
Basic Modules
Hadoop Common
Hadoop Distributed File System
(HDFS)
Hadoop YARN
Hadoop MapReduce

18. Apache Framework Basic
Modules
Hadoop Common
Hadoop Distributed File System
(HDFS)
Hadoop YARN
Hadoop MapReduce

19. Apache Framework Basic
Modules
Hadoop Common
Hadoop Distributed File System
(HDFS)
Hadoop YARN
Hadoop MapReduce

20. Apache Framework Basic
Modules
Hadoop Common
Hadoop Distributed File System
(HDFS)
Hadoop YARN
Hadoop MapReduce

26. Lecture #3
Hadoop Distributed File System
(HDFS)

27. HDFS
Hadoop Distributed File System
Distributed, scalable, and portable file-
system written in Java for the Hadoop
framework

28. HDFS

30. MapReduce Engine
Job Tracker

31. MapReduce Engine
Task Tracker
Job Tracker

32. Apache Hadoop NextGen
MapReduce (YARN)

34. What is Yarn?
• YARN enhances the power of a
Hadoop compute cluster
Scalability

35. What is Yarn?
• YARN enhances the power of a
Hadoop compute cluster
Scalability
Improved cluster utilization

36. What is Yarn?
• YARN enhances the power of a
Hadoop compute cluster
Scalability
MapReduce Compatibility
Improved cluster utilization

37. What is Yarn?
• YARN enhances the power of a
Hadoop compute cluster
Scalability
MapReduce Compatibility
Improved cluster utilization
Supports Other
Workloads

38. Lecture #4
The Hadoop “Zoo”

40. How to figure out the Zoo??

41. Original Google Stack

42. Original Google Stack

43. Original Google Stack

44. Original Google Stack
Data Integration

45. Original Google Stack

46. Original Google Stack

47. Original Google Stack

48. Facebook’s Version of the Stack
Data Integration
Coordination
Languages
Compilers
Data Store

49. Yahoo’s Version of the Stack
Data Integration
Coordination
Languages
Compilers
Data Store

50. LinkedIn’s Version of the Stack
Data Integration
Coordination
Languages
Compilers
Data Store

51. Cloudera’s Version of the Stack
Coordination
Languages
Compilers
Data Integration
Data Store

52. Hadoop Ecosystem
Major Components
Lecture #5

54. Apache Sqoop
• Tool designed for
efficiently transferring
bulk data between
Apache Hadoop and
structured datastores
such as relational
databases

56. HBASE
• Column-oriented database
management system
• Key-value store
• Based on Google Big Table
• Can hold extremely large data
• Dynamic data model
• Not a Relational DBMS

58. PIG
High level programming on top of
Hadoop MapReduce

59. PIG
The language: Pig Latin

60. PIG
Data analysis problems as data flows

61. PIG
Originally developed at Yahoo 2006

62. Pig for ETL

63. Pig for ETL

64. Pig for ETL

66. Apache Hive
• Data warehouse software facilitates
querying and managing large
datasets residing in distributed
storage

67. Apache Hive
SQL-like language!

68. Apache Hive
Facilitates querying and
managing large datasets in
HDFS

69. Apache Hive
Mechanism to project
structure onto this data and
query the data using a SQL-
like language called HiveQL

71. Oozie
Workflow scheduler system to manage
Apache Hadoop jobs

72. Oozie
Oozie Coordinator jobs!

73. Oozie
Supports
MapReduce, Pig, Apache Hive,
and Sqoop, etc.

75. Zookeeper
Provides operational services for a
Hadoop cluster group services

76. Zookeeper
Centralized service for:
maintaining configuration information
naming services
providing distributed synchronization
and providing group services

77. Zookeeper
Centralized service for:
maintaining configuration information

78. Zookeeper
Centralized service for:
maintaining configuration information
naming services

79. Zookeeper
Centralized service for:
maintaining configuration information
naming services
providing distributed synchronization
and providing group services

80. Flume
Distributed, reliable, and available
service for efficiently collecting,
aggregating, and moving large
amounts of log data

81. Additional Cloudera Hadoop
Components
Impala

83. Impala
• Cloudera's open source massively
parallel processing (MPP) SQL query
engine Apache Hadoop

84. Additional Cloudera Hadoop
Components
Spark
The New Paradigm

86. Spark
Apache Spark™ is a fast and
general engine for large-scale
data processing

87. Spark Benefits
Multi-stage in-memory primitives
provides performance up to 100 times
faster for certain applications

88. Spark Benefits
Allows user programs to load data into
a cluster's memory and query it
repeatedly
Well-suited to machine learning!!!

89. Up Next
Tour of the Cloudera’s Quick Start VM