By Rishi Arora discusses big data and Hadoop. An estimated 2,500 exabytes of new information was generated in 2012 primarily due to the internet, and the digital universe is expected to grow to 1.2 zettabytes in 2014. Hadoop is a distributed, fault tolerant, and scalable platform for big data. It consists of HDFS for storage and MapReduce for processing. HDFS stores large files across multiple nodes and provides high throughput access to application data. MapReduce allows distributed processing of large datasets across clusters of computers.

1. By :- Rishi Arora
www.rishiarora.com

9. Companies
by
estimated
Number
of
Servers

13. Source : http://www.ibmbigdatahub.com/infographic/four-vs-big-data

14. WHY BIG DATA ?
WHY NOW ?
2,500 exabytes of new information in 2012 with Internet as primary driver
Digital universe grew by 62% last year to 800K petabytes and will grow to
1.2 “zettabytes” this year
Source : An IDC White Paper- As the Economy Contracts, the Digital Universe

15. Problems
with
Big Data
?

16. Read Write Disk is Slow
1Tb Drives are read at 100Mb/sec
Use Disks in Parallel
1 HDD = 100 Mb/sec
100 HDD = 10 Gb /Sec
Solution

17. Problem #2
Hardware Failure
Single Machine Failure
Keep Multiple Copies of Data
Solution

18. Problem #3
Merge Data from Different Reads
Keep Multiple Copies of Data
Solution
Only completed results need to be taken into consideration
and failed results need to be ignored
Data needs to be compressed to be sent across the network

19. DISTRIBUTED
FAULT TOLERENT
SCALABLE
FLEXIBLE
INTILLIGENT

20. Hadoop
Components
HDFS Map Reduce
Distributed File Manager Map Reduce

21. • Designed for modest number of Large files (millions
instead of billions)
• Sequential access not Random access
• Write Once, Read Many
• Data is split into chunks and stored in multiple nodes
as blocks
• Namenode maintains the block locations
• Blocks get replicated over the data nodes
• Single namespace and accessible universally
• Computation is moved to the data – data locality
HDFS Overview

22. Map Reduce Overview
• Tasks are distributed to multiple nodes
• Each node processes the data stored in that node
• Consists of two phase:
• Map – Reads input data and output intermediate
keys and values
• Reduce – Values with the same key are sent to
the same reducer for further processing

23. HDFS
HDFS v2 YARN
ZOOKEEPER
Coordinator
FLUME
LogCollector
SQOOP
DataExchanger
Workflow
PIG
Scripting
HIVE
SQLQuery
MachineLearning
Column
Store
Hadoop Ecosystem

24. Thank You !!