Big data - Apache Hadoop for Beginner's

www.sensaran.wordpress.com
Building Big Data Analytics with
Apache Hadoop for
Beginner’s

ABOUT ME
 Myself Senthil Kumar Srinivasan
 11+ yrs. of experience in programming industry
 Working as senior consultant in capgemini
as UI Architect since 2011.
 Catch me @
 www.sensaran.wordpress.com

ROAD MAP
About Me
Targeted - Audience ?
Big-Data & Introduction ?
Big-Data’s and 4 v’s ?
About Apache Hadoop
Hadoop features
Big-Data’s and sources.
Traditional/Big- Data Approach
Quiz/Question

TARGETED AUDIENCE ?
 Professionals aspiring for a career in Big
Data analytics using Apache Hadoop.
 Analytics professionals, IT professionals,
ETL developers, project managers, and
testing novices and experts.
 Other aspirants and students looking forward to gain a thorough
understanding of the implementation of Hadoop framework.

5
 Big data is a collection of data sets that are large and complex in
nature.
 They constitute both structured and unstructured data that grow
large so fast that they are not manageable by traditional relational
database systems or conventional statistical tools.
 Today’s cheap commodity hardware, cloud architectures and open
source software bring big data processing into the reach of the less
well-resourced
WHAT IS BIG DATA ?

6
 Due to the advent of new technologies devices and social
networking sites , the amount of data produced by mankind will be
increasing every year
 Amount of data which we produced till 2003 was 5 billion gigabytes
if you pile up the data in form of disk it will acquire entire football
ground
 currently we generating same amount of data in every two days
 90% of data’s was generated in last few years
How Big is “Big Data”?

“In its raw form, oil has little value. Once processed and
refined, it helps power the world.”
—Ann Winblad

How Analysis of Big Data useful for
organization.

 Discovering what we do not know from data
 For instance , nowadays people rely on Facebook or twitter or
buying products in ebay,Flipkart,Amazon…etc
 Communicating relevant business stories from data
 Building confidence in decisions that drive business value
 Creating data for products that have business impact now

BYTES IN COMPUTER STORAGE
 Byte
 Kilobyte (KB)
 Megabyte (MB)
 Gigabyte (GB)
 Terabyte (TB)
 Petabyte (PB)
 Zettabyte (ZB)
 Yottabyte (YB)

Hobbyist
 Byte -- Byte is 8 bits.
 Kilobyte -- 1,024 bytes.
 Megabyte -- 1,024 Kilobytes.
 Gigabyte -- 1,024 Megabytes.
 Terabyte -- 1,024 Gigabytes.
 Petabyte -- 1,024 Terabytes.
 Exabyte -- 1,024 Petabytes.
 Zettabyte -- 1,024 Exabytes.
 Yottabyte -- 1,024 Zettabytes.

Hobbyist
Desktop

Hobbyist
Desktop
Internet

Hobbyist
Desktop
Internet
Big Data

Hobbyist
Desktop
Internet
Our Future

TYPES OF DATA’S

FOUR CHARACTERISTICS OF BIG DATA 4V’s

BIG DATA - VOLUME
 A typical PC might have had 10 gigabytes of storage in 2000.
 Today, Facebook ingests 500 terabytes of new data every day.
 Boeing 737 will generate 240 terabytes of flight data during a
single flight across the US.
 The smart phones, the data they create and consume; sensors
embedded into everyday objects will soon result in billions of
new, constantly-updated data feeds containing environmental,
location, and other information, including video.

BIG DATA - VELOCITY
 Clickstreams and ad impressions capture user behavior at
millions of events per second.
 High-frequency stock trading algorithms reflect market changes
within microseconds.
 Machine to machine processes exchange data between billions of
devices.
 Infrastructure and sensors generate massive log data in real-time.
 On-line gaming systems support millions of concurrent users,
each producing multiple inputs per second.

BIG DATA - VARIETY
 Big Data isn't just numbers, dates, and strings. Big Data is also
geospatial data, 3D data, audio and video, and unstructured
text, including log files and social media.
 Traditional database systems were designed to address
smaller volumes of structured data, fewer updates or a
predictable, consistent data structure.
 Big Data analysis includes different types of data.

BIG DATA - VARACITY
 The quality of captured data can vary greatly, affecting
accurate analysis.

 Web logs
 Sensor networks
 Social media
 Internet text and documents
 Internet pages
 Search index data
 Atmospheric science, astronomy,
biochemical and medical records
 Scientific research
 Military surveillance & photography archives.
BIG DATA SOURCES

LETS SEE CLEARLY
APPLICATION DATA’S
 Business apps are structured with High volume.
 It is in Structured format.
 Eg – Weather Report ,Heath care service.
 Stocks , Bank and other web service.

MACHINE DATA’S
 Companies that utilize devices that are
equipped with sensors and network
connectivity can leverage these for data as well.
 It is Semi-Structured format.
 Eg – Medical device ,Car sensor , Satellites ,
Traffic recording devices and cell towers.

SOCIAL-MEDIA DATA’S
 It is Highly unstructured.
 FB generates 10TB daily.
 Twitter generates 7TB of data
daily.

ARCHIVED DATA’S
 Archived of scanned documents, statements of
insurance forms ,medical record , paper archives
and print stream files that contain original system
of record between organization and customer.
 Highly unstructured and high in volume.

TRADITIONAL ANALYTICS APPROACH
 The requirements are defined,
followed by solution design and
build.
 Once the solution is implemented,
queries are executed.
 If there are new requirements or
queries, the system is redesigned
and rebuilt.

BIG-DATA’S APPROACH

Overview of Apache Hadoop History

ABOUT APACHE HADOOP
 Apache Hadoop ( name is derived from a cute toy elephant ) is an
open source java framework which allows you to store large amount
of data on clusters of low cost commodity hardware.
 It provides you the techniques to process the distributed data using
simple programming models.
 Hadoop Framework allows to quickly write and test the distributed
systems.

 It does not rely on hardware to provide fault
tolerance and high availability.
 It is compatible to all platforms because it is an
java based component.
 Hadoop was initially inspired by papers published
by Google outlining, its approach to handling an
avalanche of data, and has since become the de
facto standard for storing, processing and
analyzing hundreds of terabytes,and even
petabytes of data.

 Runs a number of applications on distributed
systems with thousands of nodes involving
petabytes of data.
 Has a distributed file system, called Hadoop
Distributed File System or HDFS, which enables fast
data transfer among the nodes.
WHY I NEED TO USE HADOOP ??

Scalable :
Hadoop is a highly scalable storage platform, because it can store
and distribute very large data sets across hundreds of inexpensive
servers which can be implemented in parallel way.
Economical/Cost -
In cost level comparison it is helpful to process the large set of
dats's. On other hand hadoop offers to set the large set of data for
later use it will help the major companies to reduce the cost level
implementation.
.
HADOOP FEATURES

Flexible :
It can easily process both structured and unstructured data's like
data's from social networking sites or else log processing, data
warehousing and market analysis.
Fast :
It's very efficient to process terabytes data's in a minutes and
petabytes in an hours.
Recovery/Reliable :
A key advantage of using Hadoop is its fault tolerance. When data is
sent to an individual node, that data is also replicated to other nodes
in the cluster, which means that in the event of failure, there is
another copy available for use.

READY FOR QUIZ TIME?

1 . Apache Hadoop was derived from white paper
a) Amazon
b) Google
c) Yahoo
d) All the above

2 . Four V’s in Big data denotes ?
a) Volume , Vertical , Variable , Value
b) Velocity , Volume , Variety , Varacity
c) Velocity , Variable , Value , Vertical
d) Volume, Validity, Value, Variable

3 . Which of the following aspects of refers to data size?
a) Volume
b) Velocity
c) Veracity
d) Value

4 . Which of the following is semi-structured data ?
a) Collection of tables in databases
b) Collection of text files
c) Collection of tickets
d) Collection of XML files

5 . What was Hadoop named after?
a) Creator Doug Cutting's favorite circus act
b) Cutting's high school rock band
c) The toy elephant of Cutting's son
d) A sound Cutting's laptop made during Hadoop's development

6 . Does Hadoop support both Unstructured and Structured ?
a) True
b) False

7 . All of the following accurately describe Hadoop, EXCEPT?
a) Open source
b) Real-time
c) Java-based
d) Distributed computing approach

8 . Which of the following aspects of refers to multiple data source?
a) Volume
b) Velocity
c) Velocity
d) Value

9. __________ has the world’s largest Hadoop cluster.
a) Apple
b) Datamatics
c) Facebook
d) None of the mentioned

10
Need to add

Installing Hadoop in Windows
VMware Player is a software package, offered by VMware, Inc.,
used to create and work on virtual machines
VMware Player can be downloaded free of cost from:
http://www.vmware.com/in/products/player/
Based on your system version please download 32 bit or 64 bit
version of VM ware

VMware Player—Hardware Requirements
The hardware requirements for working on VMware Player are as
follows:
>1 GHz or faster processor (2GHz recommended)
>1GB RAM minimum
>50GB of disk space to install the application

VMware Player
1.Download the required version
2. Click Next on the installation wizard.

VMware Player
Accept the default location and click Next to continue
with the installation process

VMware Player
Select the checkbox for software update – if required

VMware Player
Select the shortcut option

VMware Player
The installation wizard states that the installation
process is yet to begin. Click Continue.

VMware Player
Click Finish on the Setup Wizard Complete section. .

VMware Player
Double-click on the desktop icon for VMware Player,
accept the license agreement, and click Next.

VMware Player
We have to create the virtual machine

Please reach me
Senthil.srinivasan@capgemini.com

Big data - Apache Hadoop for Beginner's

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Big data - Apache Hadoop for Beginner's

Similar to Big data - Apache Hadoop for Beginner's (20)

More from senthil0809

More from senthil0809 (8)

Recently uploaded

Recently uploaded (20)

Big data - Apache Hadoop for Beginner's

Editor's Notes