This document provides an overview of big data, Hadoop, and related concepts: - Big data refers to large datasets that cannot be processed efficiently by traditional systems due to their size. Sources include social media, smartphones, machines, and log files. - Hadoop is an open-source framework for distributed storage and processing of large datasets across clusters of commodity hardware. It implements the MapReduce programming model. - Key Hadoop components include HDFS for storage, MapReduce for distributed processing, and related projects like Pig, Hive, HBase, Flume, Oozie, and Sqoop. Companies use Hadoop for applications involving large datasets, such as log analysis, recommendations, and business intelligence

1. SOFTWARE DEVELOPMENT DONE RIGHT
Netherlands | USA | India | UK | France

2. What is Big
Data?

Generally refers to data that can not be
processed by traditional systems efficiently
mainly because of it's size.

Twitter/Facebook example

Facebook – 500TB data daily

Twitter – 250million tweets daily

90% of data has been generated in last 2-3
years.

3. Big Data
Sources

Sources -
• Social networking sites like twitter, facebook etc.
• Smart phones
• Trading platforms
• Machines
• Log Files

This data is used for different purposes like
• Product Trends
• Market Analysis

4. What is
Hadoop ?

Apache Hadoop is a Framework for running
applications on large cluster built of commodity
hardware.

Transparently provides applications both
reliability and data motion.

Implements a computational paradigm named
Map/Reduce where application is divided in
small fragments of work.

Provides a distributed file system (HDFS)

Transfers code near to data.

Hadoop opened the gates for processing Big
Data

5. Hadoop's
History

Hadoop is based on work done by Google

GFS – HDFS

Google Map Reduce – Hadoop Map
Reduce

BigTable – HBase

6. Hadoop
Features

Partial Failure Support

Data Recoverability

Component Recovery

Consistency

Scalability

7. Hadoop
Components

Core Components
• HDFS – Hadoop Distributed File System
• Map Reduce

Projects in Hadoop Ecosystem
• Pig, Hive, HBase, Flume, Oozie, Sqoop
etc.

8. HDFS

9. Map/Reduce

10. Case
Study

Product - Data Quality and cleansing product
solutions.

Before Hadoop

Two node DB cluster

Multi-threaded java application for de-
duplication

1 million records took 10 hrs. to process

After Hadoop

8 GB Ram, 4 cores, 4 machines in cluster.

1 million records took 30 min to process

11. Hadoop In
Use

Any application which has

> 10TB data

Needs fast and cheap processing

Log Analysis

Recommendation Engine

Feed Analysis

Data Mining

Statistical Analysis

ETL Processing

Business Intelligence

12. Cloudera

Cloudera is “The commercial Hadoop
company”.

Founded by leading experts on Hadoop
from Facebook, Google,Oracle and Yahoo.

Provides consulting and training services
for Hadoop users.

Staff includes committers to virtually all
Hadoop projects.

13. Resources

Books

Hadoop : The Definitive Guide (by Tom White)

Hbase : The Definitive Guide (by Lars George)

MapReduce Design Patterns (by Donald Miner)

Web

http://hadoop.apache.org/

http://hbase.apache.org/

http://research.google.com/archive/bigtable.html

http://research.google.com/archive/mapreduce-osdi04.pdf

14. Contact us @
Xebia India
Website
www.xebia.com Thought Leadership
www.xebia.in http://blog.xebia.com
www.xebia.fr http://podcast.xebia.com