This document provides an overview of big data and Hadoop. It defines big data as large, complex datasets that are difficult to store and process using traditional systems. Examples of big data sources are listed. Hadoop is introduced as an open source framework for distributed processing of large datasets across commodity computers. Key components of Hadoop like HDFS for data storage and MapReduce for parallel processing are described. Reasons for moving to Hadoop include its ability to handle large, unstructured datasets across clusters of servers in a scalable way. The future of data growth and the Hadoop ecosystem are also discussed briefly.

1. Demystifying
Big Data
Hadoop
&

2.  What is Big Data
 Case Study – Retailer
 Why DFS
 What is Hadoop
 Brief history
 Core Components
 Limitations
 Ecosystem
 Why move to Hadoop
 Future of Data
 Oracle Big Data Appliance

4. Big Data refers to collection of
data sets so large and complex
that its difficult to store and
process using traditional data
management systems

5. Huge data source
Social Network
Online Shopping (Clickstream data)
Airlines, Banking
CRM, Stock Exchange
Hospital, Hospitality
Geo Location data
Sensors, CCcams
Government data

6. If we are living in 100% of data world then
90% of data is generated in last 3 years

8. Un-Structured Data is Exploding

11. Case Study : Retail

12. Limitation of Existing Architecture

13. Solution: A Combined Storage Layer

14. Why DFS ?

15. Why DFS ?
1000,000 MB/(100*4*60)
41.67 min

16. Hadoop is a framework that
allows for distributed processing
of large data sets across clusters of
commodity computers using a
simple programming model

17. Hadoop was designed to enable applications to make most out of cluster
architecture by addressing two key points:
1. Layout of data across the cluster ensuring data is evenly distributed
2. Design of applications to benefit from data locality
It brings us two main mechanism of hadoop, hdfs and hadoop MapReduce

18. Hadoop Core Components

19. Splits, Scatter, Replicate and manage data across nodes

20. MapReduce is a mechanism to execute an application in parallel
• by dividing it into tasks
• collocating these task with part of the data
• collecting and redistributing intermediate results
• and managing failures in all nodes of the clusters

21. Hadoop Key Characteristics
• Open source (Apache License)
• Large unstructured data sets (Petabytes)
• Simple Programming model running on Linux
• Scalable from single server to thousands of machines
• Runs on commodity hardware and the cloud
• Application level fault tolerance
• Multiple tools and libraries

22. Hadoop History

23. Hadoop can handle small datasets but you can’t unleash the power of
hadoop.
There is overhead associated with each data distribution. If dataset is small
you won’t get huge advantage in hadoop.
If dataset is small and unstructured, you will try to collate the data.
Areas where Hadoop is not good fit Today

24. Hadoop Ecosystem

26. Why move to Hadoop

27. future

28. Oracle Big Data Appliance