This document provides an overview of a training program on Big Data and Hadoop. The training includes live online classes, recorded class materials, quizzes and assignments. Key topics covered include Hadoop architecture, MapReduce, YARN, Pig Latin, Hive, HBase and project work. The training aims to help students understand Big Data challenges, how Hadoop addresses them and gain skills required for jobs working with Big Data.

1. Big Data & Hadoop

2. ❑ LIVE On-Line Classes
❑ Class recordings made available for life time
❑ Quizzes and Assignments at end of each chapter
❑ Technical support
❑ Project work
❑ Assessment and Certification
❑ Post Training Guidance and Support
❑ Assistance in finding relevent Job

3. Day 1 Day 2
Week 1 Understanding Big Data
Hadoop Architecture
Hadoop Cluster
Data Loading Techniques
Week 2 Basic MapReduce Advanced MapReduce
YARN 2.0
Week 3 PIG Latin Hive
Week 4 NoSQL Databases, HBase and
ZooKeeper
Project Work

4. ❏
❏
❏
❏
❏
❏
❏
❏
❏
❏

5. ❏
❏
❏
❏
NYSE generates about one
terabyte of new trade data per
day to Perform stock trading
analytics to determine trends for
optimal trades.

6. Volume Variety Velocity
Big Data

7. ❏
❏
❏
❏

8. ❏
❏
❏
❏

9. ❏
❏
❏
❏
❏

10. ❏
❏
❏
❏
❏
❏
❏
❏

11. ❏
❏
❏
❏
❏
❏
❏
❏

12. ❏
❏
❏
❏
❏
❏
❏
❏
❏

15. ❏
❏

17. Storage - Backup /
Read - Write
Processing
(ETL)
Usage /
Visualization

18. OLTP
RDBMS
Soci
al
Logs
Expensive
Storage and
processing
Lot of Data
Discarded
Storage spread across.
Not easily accessible.
Limited storage capacity
Reports
Reports

19. OLTP
RDBMS
Soci
al
Logs
Lot of Data
Discarded
Reports
(Batch)
Hadoop
DW Reports

20. ❏
❏
❏
❏
❏
❏

21. ❏
❏

26. 1 Machine
4 I/O Channels
Each Channel -- 100 MBps
100 Machines
4 I/O Channels
Each Channel -- 100 MBps

27. 1 Machine
4 I/O Channels
Each Channel -- 100 MBps
100 Machines
4 I/O Channels
Each Channel -- 100 MBps
Reading 1 TB Data
45 Minutes .45 Minutes

28. Story of Hadoop
❏
❏
❏
❏
❏
❏
❏
❏
❏
❏
❏
❏

29. ❏
❏

30. ❏
❏
❏

31. ❏
❏
❏
❏
❏
❏
❏

32. Characterstics of Hadoop
Hadoop
Reliable
Economical
Scalable
Fault Tolerant

35. ❏
❏
❏
❏
❏
❏
❏
❏

36. Hadoop Core Components

39. ❏
❏
❏
❏
❏

40. ❏
❏
❏
❏
❏
❏
❏
❏
❏
❏
Name Node:
Keeps track of overall file
directory structure and the
placement of Data Block
Name Node
(Stores metadata only)
METADATA:
/user/doug/hinfo-> 1 3 5
/user/doug/pdetail-> 4 2

41. NameNode
Edit Logs
FSImage

42. ❑
❑
❑
❑
NameNode
Sedondary
Namenode
File System
Metadata
Its been
an hour
?

43. Quiz

44. Quiz

45. Quiz
When the NameNode fails, Secondary NameNode takes over instantly and
prevents Cluster Failure:
❑ TRUE
❑ FALSE

46. Quiz
When the NameNode fails, Secondary NameNode takes
over instantly and prevents Cluster Failure:
❑ TRUE
❑ FALSE
False. Secondary NameNode is used for creating NameNode
Checkpoints. NameNode can be manually recovered using ‘edits’ and
‘FSImage’ stored in Secondary NameNode.

47. JobTracker

48. JobTracker (cotd..)

49. JobTracker (cotd..)

50. Quiz

51. Quiz

54. Rack 1 Rack 2 Rack 3
Block A Block B Block C
Topology script property topology.script.file.name in core-site.xml

55. ❑
❑

56. ❑
❑

60. Green - GA Versions
Black - Not Released by
Apache yet
Red - Commercial

61. ❏ http://blog.cloudera.com/blog/2012/01/an-update-on-apache-
hadoop-1-0/
❏ https://blogs.apache.org/bigtop/entry/all_you_wanted_to_know
❏ https://hadoop.apache.org/releases.html
❏ http://hortonworks.com/blog/apache-hadoop-2-is-ga/

62. ❏
❏
❏
❏

63. Class 2 Pre-work
❏ Setup hadoop environment using documents provided on google
drive
❏ CDH3 (recommended) or CDH4
❏ Execute basic linux commands
❏ Execute HDFS hands on commands
❏ Attempt the class-1 assignment

64. Thank You !
See you in next class